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Abstract 

All  organizations  face  the  continuous  challenge  of  a  dynamic  and  ever-changing 
operations  environment.  They  must  adapt  to  new  paradigms  quickly  or  end  up  on  the 
road  to  obsolescence.  Never  has  this  been  truer  for  the  U.S.  Air  Force  than  in  the  2 1st 
century.  The  logistics  organization  of  the  Air  Force  supports  a  worldwide,  24/7 
operation,  executing  the  national  directive  of  US  Policy.  Past  logistics  operation  policies 
have  now  been  proven  to  no  longer  be  sufficient  to  meet  the  needs  of  the  war-fighter. 
Shrinking  budgets,  aging  equipment,  and  the  austere,  disparate  operating  locations 
demand  sweeping  changes  in  how  the  logistic  machine  operates.  The  Expeditionary 
Combat  Support  System  (ECSS)  is  the  Air  Force  wide  Enterprise  Resource  Planning 
(ERP)  system  designed  to  tackle  these  very  challenges.  An  ERP  implementation 
endeavor  consists  of  a  large  number  of  critical  areas  that  have  to  be  addressed. 
Management  Support,  Business  Process  Reengineering,  Strategy  and  Governance 
modeling,  and  Legacy  Systems  Evaluation  and  Conversion  are  just  a  few  of  the  key  areas 
that  need  to  be  managed  successfully  for  effective  ERP  implementation.  This  study 
focuses  on  one  area,  Legacy  Systems  Evaluation  and  Conversion.  This  study  explores 
the  transition  of  28  current  Air  Force  Maintenance  metrics  into  the  Oracle  ERP  software 
platform.  Evaluation  of  these  metrics  by  operational  maintenance  managers  provides 
insight  into  the  importance  and  effectiveness  of  the  current  metrics  as  well  as  the  clarity 
and  potential  success  for  translating  the  proposed  new  metrics  for  the  ECSS  program. 
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Evaluation  of  Air  Force  Aircraft  Maintenance  Metrics  for  Integration  into  the 


Expeditionary  Combat  Support  System 


I.  Introduction 

Background 

All  organizations  face  the  continuous  challenge  of  a  dynamic  and  ever-changing 
operations  environment.  They  must  adapt  to  new  paradigms  quickly  or  end  up  on  the 
road  to  obsolescence.  Never  has  this  been  truer  for  the  U.S.  Air  Force  than  in  the  2 1st 
century.  The  logistics  organization  of  the  Air  Force  supports  a  worldwide,  24/7 
operation,  executing  the  national  directive  of  US  Policy.  Past  logistics  operation  policies 
are  no  longer  be  sufficient  to  meet  the  needs  of  the  war-fighter.  Shrinking  budgets,  aging 
equipment,  and  the  austere,  disparate  operating  location  demand  sweeping  changes  in 
how  the  logistic  machine  operates.  Expeditionary  Combat  Support  System  (ECSS)  is  the 
Air  Force  wide  Enterprise  Resource  Planning  (ERP)  system  designed  to  tackle  these  very 
challenges. 

ECSS  will  replace  over  250  legacy  logistics  support  systems  currently  in  operation.  The 
Air  Force  has  selected  the  Oracle  Software  Suite  as  the  vehicle  of  choice  to  provide  the 
platfonn  to  support  this  ERP  effort.  As  ECSS  moves  closer  to  execution,  many 
fundamental  questions  arise  on  how  to  effectively  transition  from  the  current  legacy 
systems  to  the  Commercial-Off-The-Shelf  (COTS)  provided  framework. 
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One  of  the  cornerstone  programs  of  the  logistics  machine  is  the  Air  Force  Aircraft 
Maintenance  Metrics  Program.  The  mission  of  the  United  States  Air  Force  is  to  fly,  fight 
and  win.. .in  air,  space  and  cyberspace.  (Air  Force,  2003).  In  order  to  achieve  this 
mission,  the  Air  Force  must  fly  aircraft,  but  to  fly  aircraft  it  has  to  be  able  to  perform 
maintenance.  Maintenance  managers  must  know  how  well  maintenance  is  being 
performed  in  order  to  generate  the  aircraft  needed  to  accomplish  the  core  mission.  The 
U.S.  Air  Force  has  established  a  comprehensive  framework  of  perfonnance  measures  in 
order  to  manage  its  maintenance  activities. 

Problem  Statement 

At  this  time  it  is  not  known  if  any  of  the  current  process  control  measures  have  any 
comparison  with  those  established  in  the  Oracle  ERP  suite.  This  knowledge  will  be 
critical  to  the  success  of  ECSS,  since  one  of  the  fundamental  concepts  of  ERP 
implementation  is  to  avoid  creating  any  customized  program  code  beyond  what  is 
provided  in  the  ERP  software. 

Research  Objectives 

The  objective  of  this  research  is  to  recommend  both  a  method  to  effectively  integrate  the 
current  AF  Aircraft  Metrics  Program  requirements  into  the  Expeditionary  Combat 
Support  System  as  well  as  recommend  which  metrics  should  be  integrated.  To 
accomplish  this,  the  study  developed  a  comparison  framework  between  current  metrics  in 
use  today  and  the  best-practice  metrics  provided  in  the  Supply  Chain  Organization 
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Reference  model  and  the  Oracle  software  suite.  Researchers  then  leveraged  subject 
matter  expert  evaluations  of  the  metrics  in  order  to  detennine  suitability  for  integration. 

Research  Questions 

In  order  to  provide  focus  and  direction  for  the  study  two  research  questions  were 
proposed: 

1.  How  well  do  current  Air  Force  maintenance  metrics  translate  into  the  proposed 
Expeditionary  Combat  Support  System  Enterprise  Resource  Planning  framework? 

2.  Which  current  Air  Force  maintenance  metrics  should  be  translated  into  the 
proposed  Expeditionary  Combat  Support  System  Enterprise  Resource  Planning 
framework? 

Investigative  Questions 

Four  investigative  questions  were  derived  from  the  initial  research  questions: 

1 .  How  important  is  the  current  metric  for  managing  aircraft  maintenance? 

2.  How  effective  is  the  current  metric  in  managing  aircraft  maintenance? 

3.  How  adequately  does  the  proposed  metric  replace  the  current  metric? 

4.  How  easily  understood  is  the  new  proposed  metric? 

These  investigative  questions  were  used  to  evaluate  Air  Force  maintenance  metrics  as 
seen  in  Appendix  A. 
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Methodology 

A  qualitative  research  methodology  was  applied  to  this  research  problem.  Specifically  a 
first-degree  multi-case  study  analysis  was  performed  to  evaluate  26  current  Air  Force 
metrics  and  2  SCOR  metrics  and  then  develop  a  translation  framework.  Subject  matter 
experts  were  interviewed  and  data  compiled  to  validate  the  effectiveness  of  the 
comparison  framework  and  evaluate  the  importance  and  effectiveness  of  the  current 
metrics. 

Implications 

One  of  the  key  critical  successes  of  ERP  implementation  is  effective  translation  of 
current  legacy  performance  systems  into  the  selected  ERP  program.  This  study  has 
developed  a  method  that  can  be  used  to  translate  current  aircraft  maintenance  to  the 
available  metrics  in  the  new  system.  However,  this  is  but  one  system  of  measures  used 
by  the  Air  Force  logistics  community.  This  framework  can  also  be  replicated  in  order  to 
integrate  other  perfonnance  measure  programs  into  the  ECSS  architecture. 

Summary 

The  Expeditionary  Combat  Support  System  is  one  of  the  cornerstones  of  logistics 
transfonnation  in  the  US  Air  Force.  The  establishment  of  an  Enterprise  Resource 
Planning  program  will  allow  future  logisticians  to  better  understand  and  efficiently 
manage  logistics  processes.  An  ERP  implementation  endeavor  consists  of  a  large 
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number  of  critical  areas  that  have  to  be  addressed.  However,  this  study  will  focus  on  one 
area  -  the  integration  of  the  Air  Force  Aircraft  Maintenance  Metrics  Program  into  ECSS. 

The  first  chapter  of  this  study  has  outlined  the  impetus  and  direction  of  the  research  to  be 
discussed.  Chapter  2  will  provide  an  in  depth  review  of  the  relevant  literature  covering 
the  problem  statement  and  research  questions.  Chapter  3  will  outline  the  methodology 
framework  and  establish  the  criteria  for  answering  the  research  question,  while  Chapter  4 
discusses  the  collected  and  analyzed  data.  Chapter  5  will  summarize  the  conclusions  and 
results  of  the  research  analysis  and  provide  any  relevant  limitations  discovered. 
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II.  Literature  Review 


Overview 

An  old  adage  states  “The  only  constant  in  life  is  change”.  Never  is  this  truer  than  in  the 
world  of  logistics  management.  All  organizations  face  the  continuous  challenge  of  the 
dynamic  and  ever  changing  operations  environment.  They  must  adapt  to  new  paradigms 
or  end  up  on  the  road  to  obsolescence.  Never  has  this  been  truer  for  the  U.S.  Air  Force 
than  in  the  21st  century.  The  logistics  organization  of  the  Air  Force  supports  a 
worldwide,  24/7  operation,  executing  the  national  directive  of  US  Policy.  Past  logistics 
operation  policies  are  no  longer  sufficient  to  meet  the  needs  of  the  future  war-fighter. 
Shrinking  budgets,  aging  equipment,  and  austere,  disparate  operating  location  demand 
sweeping  changes  in  how  the  logistic  enterprise  operates. 

One  way  these  organizations  are  adapting  to  this  need  for  change  is  by  implementing 
Enterprise  Resource  Planning  (ERP)  programs.  In  order  to  facilitate  effective  ERP 
systems,  organizations  must  be  able  to  convert  their  current  operation  systems  into  their 
new  ERP  programs.  This  literature  review  will  outline  the  drive  for  change  in  the 
Department  of  Defense  and  its  effect  on  future  US  Air  Force  operational  planning.  These 
directives  are  the  purpose  behind  the  Air  Force  developing  an  enterprise  wide 
managements  system.  The  review  will  then  discuss  the  critical  success  for  ERP 
implementation.  Critical  success  factors  must  be  effectively  addressed  in  order  for  an 
ERP  implementation  to  be  successful.  The  final  sections  will  discuss  the  characteristics 
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of  logistics  process  measures  and  review  the  Air  Force’s  Aircraft  Maintenance  metrics 
program  that  provides  guidance  and  feedback  to  Air  Force  maintenance  managers. 


Transformation  and  the  United  States  Air  Force 

It  has  been  estimated  that  the  Air  Force  spends  20  to  30  Billion  dollars  on  current  supply 
chain  processes  supporting  the  war  fighter  (CSC,  2007).  Air  Force  logisticians  operate 
24  hours  a  day  7  days  a  week,  on  every  continent  on  the  globe.  They  support  Anny, 
Navy,  Marine  and  other  government  agencies  across  the  full  spectrums  of  military 
operations.  Air  Mobility  Command,  primary  provider  of  strategic  airlift  capability,  have 
aircraft  flying  24  hrs  a  day,  365  days  a  year  from  anywhere  to  frozen  Antarctica  to  120 
degree  Iraq.  This  is  a  truly  monumental  operation  that  is  coordinated  daily,  sometimes 
even  hourly  by  the  dedicated  men  and  women  serving  in  all  logistics  capacities. 


In  2003  Department  of  Defense  Secretary  Donald  Rumsfeld  laid  out  the  Trans  formation 
Planning  Guidance  (TPG),  a  comprehensive  roadmap  to  transition  the  US  Military  from 
the  industrial  age  to  the  infonnation  age  (DoD,  2003).  DoD  TPG  outlined  the  reason  for 
Transformation  as  follows: 

Transformation  is  necessary  to  ensure  U.S.  forces  continue  to  operate  from  a  position 
of  overwhelming  military  advantage  in  support  of  strategic  objectives.  We  cannot 
afford  to  react  to  threats  slowly  or  have  large  forces  tied  down  for  lengthy  periods. 
Our  strategy  requires  transformed  forces  that  can  take  action  from  a  forward  position 
and,  rapidly  reinforced  from  other  areas,  defeat  adversaries  swiftly  and  decisively 
while  conducting  an  active  defense  of  U.S.  territory.  Transformed  forces  also  are 
essential  for  deterring  conflict,  dissuading  adversaries,  and  assuring  others  of  our 
commitment  to  a  peaceful  world.  Over  the  long  term,  our  security  and  the  prospects 
for  peace  and  stability  for  much  of  the  rest  of  the  world  depend  upon  the  success  of 
transformation. 
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In  2003,  in  direct  response  to  the  Secretary’s  charge  the  Air  Force  leadership  presented 
its  own  plan  for  transformation,  the  Transformation  Flight  Plan.  The  Flight  Plan  is  a 
reporting  document  and  program  in  order  to  fully  capitalize  on  the  knowledge  and 
innovation  within  the  United  Stated  Air  Force  and  translate  its  efforts  directly  into  the 
TPG  (Air  Force,  2004).  Following  the  Flight  Plan,  Air  Force  leadership  codified  the 
transformation  directive  into  Air  Force  Smart  Operations  for  the  21st  Century  (AFS021). 
Under  the  purview  of  AFS021,  the  Air  Force  Headquarters  Department  of  Installation 
and  Logistics  established  the  Expeditionary  Logistics  for  the  21st  Century  (ELog21) 
program  to  guide  and  direct  logistics  transformation  within  the  logistics  community  (Air 
Force,  2004).  ELog21  is  the  primary  logistics  transformation  program  focused  on 
creating  leaner,  more  lethal  combat  support  capabilities  in  order  to  fully  capitalize  on 
current  and  future  air  and  space  resources  (Bergdolt,  2007). 

ELog21  focuses  on  four  desired  areas  of  effect:  Enterprise  View,  Integrated  Processes, 
Optimized  Resources,  and  Integrated  Technology.  In  order  to  achieve  these  interrelated 
end-states,  the  ELog2 1  program  has  established  over  20  transformation  initiatives  to 
fundamentally  change  the  current  logistics  support  environment  (Dunn,  2007). 
Unfortunately,  the  system  in  which  the  dedicated  logistics  professionals  operate,  while 
not  broken,  does  not  provide  the  necessary  capability  to  meet  the  desired  end  state.  The 
system  maintains  legacy  programs  born  out  of  a  bygone  area  of  military  process  control. 
Air  Force  logistics  processes  are  reactionary,  functionally  stove-piped  operating  from 


over  400  disparate  information  systems  (Dunn,  2007).  In  order  to  create  a  new  paradigm 
of  logistics  capability  and  thought,  a  new  system  has  to  be  initiated  to  fully  integrate  the 
logistics  enterprise,  the  Expeditionary  Combat  Support  System. 


Expeditionary  Combat  Support  System 

The  Expeditionary  Combat  Support  System  (ECSS)  is  the  infonnation  technology 
baseline  for  logistic  systems  modernization.  Table  1  outlines  the  twelve  core  capabilities 
that  ECSS  will  provide  to  the  future  logistics  community. 

Table  2.1.  ECSS  Capabilities  (Bergdolt,  2007) 


Capability 

Advanced  Planning  and 
Scheduling 

Demand  forcasting  and  collaborate  plans  development 

Material  Management, 
Contracting  and  Logistics 
Finance 

Procurement  and  purchasing,  contract  management,  repair  and 
maintenance  support,  and  finance  transactions 

Configuration  and  Bill  of 
Materials 

Primary,  Alternate,  common  and  phantom  planning  and 
confiquation  BOMs 

Repair  and  Maintenance 

Planning  and  operations,  visibility  into  maintenace  costs, 
equipment  history,  and  mainteainability  and  reliability  data 

Product  Life-Cycle 
management 

Integrated  enginneering  and  execution  functions;  life-cycle  view  of 

assets 

Customer  Relationship 
Management  and  Order 
Management 

Order  fulfillment  processes  and  tracking  form  material  requests  to 
fulfillment  of  the  order 

Distiribution  and 
Transportation 

Physical  control  of  material  to  include  cycle  counting,  storage, 
shipping,  transporting,  and  tracking 

Facilities  Management 

Track  and  maintain  equipment;  provide  asset  visability 

Quality  Control 

Data  collection;  reporting  with  traceability  back  to  transaction; 
trend  analysis 

Document  Management 

Identify  and  maintain  documents  use  in  current  and  future  state 
processes;  data  cleansing  standardizes  formats  and  methods  used  to 

link  data 
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The  12  capabilities  outlined  in  Table  2. 1  will  be  the  end  result  of  the  migration  of  the  400 
legacy  systems  currently  in  place.  ECSS  is  at  its  core  an  Enterprise  Resource  Planning 
program  built  around  a  Commercial-off-the-Shelf  (COTS)  software  suite.  By  effectively 
converting  current  legacy  process  requirements  to  the  COTS  best-business  practices 
established  within  the  software,  true  logistics  transformation  breakthroughs  may  be 
achieved  (Cain,  2007). 

Enterprise  Resource  Planning 

Enterprise  resource  planning  (ERP),  like  all  innovation,  was  born  out  of  necessity.  The 
business  world  had  experienced  a  major  paradigm  shift.  Gone  was  the  era  of  single 
product  factories,  large  unskilled  labor  pools  and  simple  logistics  process  management. 
Today’s  business  world  is  dynamic  and  ever  changing.  Multi-product  corporations,  ever 
changing  mergers,  hostile  takeovers,  and  global  consumer  market  operations  demand  a 
total  integrated  planning  system. 

Blackstone  and  Cox  (2005)  defined  ERP  as  a  “framework  for  organizing,  defining,  and 
standardizing  the  business  processes  necessary  to  effectively  plan  and  control  an 
organization  so  the  organization  can  use  its  internal  knowledge  to  seek  external 
advantage”.  Most  ERP  system  frameworks  focus  on  the  manufacturing  planning  and 
control,  integrating  finance  and  accounting,  human  resources,  payroll,  and 
sales/marketing  with  production  and  distribution  (Jacobs  and  Weston,  2006).  Essentially 
an  ERP  system  provides  a  unified  interface  across  the  entire  enterprise  (Davenport,  1998) 


10 


The  evolution  of  the  modern  day  ERP  architecture  began  in  the  early  1960’s.  The 
development  of  commercial  computer  mainframes  and  the  introduction  of  the  Reorder 
point  (ROP)  systems  put  efficient  manufacturing  planning  and  control  programs  (MPC) 
in  the  hand  of  industry.  However  these  systems,  dubbed  Material  Requirements  planning 
(MRP),  were  large,  resource  intensive  and  expensive,  necessitating  a  large  investment  by 
the  manufacture  industry  (Fawcett,  Ellram,  and  Ogden,  2007). 

The  next  revolution  occurred  in  the  late  1970s.  Business  strategy  shifted  away  from 
cost  minimization  to  more  of  a  marketing  focus  creating  emphasis  on  production 
integration  and  planning  (Jacobs  and  Weston,  2007).  This  shift  also  was  driven  forward 
by  the  fielding  of  IBM’s  COPICS  software  and  the  Model  360  mainframe  computer  with 
higher  capacity  random  access  storage.  This  shrunk  the  support  footprint  and  allowed  the 
development  of  third  party  programming  software  by  companies  such  as  Oracle,  J.D. 
Edwards  and  Lawson  software. 

In  the  1980s,  low  cost  mainframes  and  the  introduction  of  flexible  disk  drive  made  MRP 
systems  available  to  medium  to  small  manufacturing  enterprises.  Designated  MRP-II, 
manufacturing  resource  planning  II,  the  newer  systems  provided  more  functions  and 
capability  than  the  previous  MRP  system  (Fawcett,  Ellram,  and  Ogden,  2007)..  The  rise 
of  quality  management  during  this  time  also  played  a  large  role  in  the  continuing 
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evolution  of  manufacturing  control  systems,  further  widening  the  scope  of  control  these 
systems  provided  to  the  industry 

1990  saw  the  first  true  designation  of  the  ERP  nomenclature,  these  early  ERP  systems 
replaced  several  legacy  systems  within  an  enterprise  (Venkatachalam,  2006).  New 
unified  database  architecture  was  provided  by  single  vendors,  such  as  SAP,  Oracle  or 
J.D.  Edwards,  increasing  the  capability  of  total  integration  significantly.  One  key 
advantage  of  the  new  systems  was  the  incorporation  of  open  architecture  programming, 
which  allowed  third-party  development  companies  to  integrate  seamlessly  (Rainer  and 
Turbin,  2008).  This  characteristic  in  conjunction  with  the  year  2000  data  problem 
created  a  dramatic  growth  of  ERP  implementation  within  both  large  and  small 
manufacturing  industry. 

ERP  Critical  Success  Factors 

As  with  the  implementation  of  any  new  system,  failure  is  as  much  a  companion  as 
success.  The  key  to  achieving  the  latter  while  avoiding  the  fonner  lies  in  careful 
planning  and  identifying  the  key  areas  for  focused  management  attention. 

Bullen  and  Rockart  (1986)  defined  these  areas  in  ERP  implementation  as  Critical  Success 
Factors  (CSF).  Specifically,  they  define  “the  limited  number  of  areas  in  which 
satisfactory  results  will  ensure  successful  competitive  perfonnance  for  the  organization.” 
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(Bullen  an  Rockart,  1986)  Essentially  CSFs  are  the  areas  where  “things  must  go  right” 
for  the  ERP  goals  to  be  realized  (Ngai,  2008) 

A  common  mantra  of  business  is  anything  can  be  done  with  enough  money  and  time,  but 
most  of  the  time  businesses  have  neither  enough  time  nor  enough  money.  This  is 
especially  true  of  ERP  implementation.  Over  90  percent  of  companies  that  have 
implemented  an  ERP  system  failed  on  the  first  time.  Failure  of  an  ERP  implementation 
was  defined  as  a  company  not  achieving  their  original  stated  goals  for  cost,  utilization 
and  expected  perfonnance  improvement  (Sun  et  al,  2005).  Compound  that  figure  with 
the  fact  that  by  the  end  of  2008  over  15.8  billion  dollars  will  be  spent  on  ERP  programs 
(Ehie,  2005),  successful  implementation  is  absolutely  critical.  However  while  the 
literature  reveals  extensive  study  into  defining  CSFs,  there  is  little  consistent  agreement 
on  the  number  of  CSFs  required  or  the  order  of  importance.  Ngai  et  al  (2008)  has 
developed  a  framework  of  18  factors,  in  contrast  Sun  et  al  (2005)  has  only  outlined  5  core 
CSFs  as  necessary  for  implementation  success. 

Table  2.2  represents  the  predominant  CSFs  as  defined  in  the  reviewed  literature.  While 
each  author  has  titled  each  CSF  differently  the  categories  outlined  below  represent  the 
common  definition  used  by  sources  reviewed. 
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Table  2.2.  Rank  Order  of  CSF  Importance 
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Based  on  the  Table  2,  the  top  six  CSFs  for  ERP  implementation,  in  order  of  importance, 
are  1.  Management  Support,  2.  Legacy  System  Evaluation  and  Conversion,  3.  Strategy 
and  Governance  Model,  4.  Business  Process  Reengineering,  5.  Change  Management, 
and  6.  Implementation  Teamwork  and  Composition.  While  a  business  enterprise 
implementing  an  ERP  needs  to  focus  on  all  the  CSFs,  the  reality  dictates  a  organization 
does  not  have  enough  resources  to  focus  on  all  areas  equally.  This  framework  will  allow 
the  ERP  implementation  team  to  effectively  prioritize  time  and  resources  to  get  the  most 
out  of  their  efforts.  While  all  of  these  factors  are  critical  to  implementation  success  this 
study  will  focus  on  Legacy  Systems  Evaluation  and  Conversion,  specifically  how  to 
translate  current  Air  Force  metrics  into  the  ECSS  system. 
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Process  Measurement  and  Evaluation 


Implementing  an  ERP  can  be  a  monumental  undertaking.  Organizations  devote  great 
time,  effort  and  money  to  establishing  their  systems.  One  of  the  critical  success  factors 
indicated  previously  is  Legacy  System  Evaluation  and  Conversion.  Essentially,  it  means 
codifying  how  an  enterprise  operates,  then  effectively  correlating  those  operating 
measures  into  the  new  ERP  product  suite.  This  requires  a  solid  understanding  of  what 
and  how  an  organization  tracks  their  operation  processes. 

Every  organization  has  a  purpose;  otherwise  it  would  not  be  an  organization.  According 
to  the  Encyclopedia  Britannica  (2008),  an  organization  is  defined  as  “an  entity  fonned  for 
the  purpose  of  carrying  on  commercial  enterprise”.  In  order  to  accomplish  its  purpose  a 
logistics  organization  has  to  know  where  it  has  been  and  where  it  needs  to  go.  In  other 
words  it  has  to  be  able  to  effectively  measure  its  performance  in  delivering  its  product  or 
service  (Keebler  et  al.,  1999).  Performance  metrics  represent  critical  elements  in 
translating  an  organizations  strategy  across  the  enterprise;  they  link  the  individual 
behavior  to  the  organizational  goals  (Boyd  and  Cox,  1997). 

Performance  measures  or  metrics  are  a  verifiable  measure,  quantitative  or  qualitative  in 
nature,  defined  with  respect  to  a  fixed  maintenance  point  (Melnyk  et  al.,  2004)  However 
in  this  era  of  high  technology  and  fast  paced  business  world,  there  are  several  challenges 
to  the  effective  and  appropriate  use  of  metrics.  Some  of  the  more  prominent  challenges 
to  industry  are:  increasing  needs  to  manage  the  total  supply  chain,  shrinking  product  life- 
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cycles,  huge  quantities  of  data  that  lack  quality,  and  new  opportunity  alternatives  and 
markets.  All  this  leads  to  a  need  for  companies  to  critically  evaluate  their  metrics 
programs. 

There  are  hundreds  of  metrics  available  to  managers  today  (Griffis  et  al.,  2004).  This 
means  the  hard  job  of  any  manager  is  not  choosing  a  metric  but  choosing  the  right  metric 
for  their  production  process.  At  its  most  basic  level  metrics  should  provide  two  things, 
they  should  be  meaningful  and  meet  the  organizational  needs.  While  these  two  ideas 
may  seem  to  be  common  sense,  often  companies  regularly  violate  them.  One  way 
companies  deviate  from  these  premises  is  they  consult  the  wrong  metrics  for  the  process 
being  evaluated  (Griffis  et  al.,  2004).  A  plant  manager  has  no  use  for  financial  based 
metrics  to  evaluate  how  his  plant  is  running,  likely  as  not,  he  cannot  control  or  influence 
them  effectively  anyway  (Beiscel  and  Smith,  1991).  Another  common  pitfall  of  metrics 
selection  is  when  organizations  use  proxy  measures  selected  for  convenience,  but  which 
are  often  unrelated  to  true  division  performance  (Tsang,  1999). 

As  defined  by  Callahan  (2007),  meaningful  metrics  meet  a  specific  objective  for  an 
organization.  Metrics  must  be  measured  against  a  clear  standard,  a  set  or  range  of  values 
that  give  direction  to  the  metric  user.  Metrics  also  must  be  meaningful  in  how  they 
measure  not  just  what  processes  they  measure.  Tracking  customer  satisfaction  thru 
On-Time-Delivery  of  orders  may  be  a  meaningful  goal.  However  if  the  measurement  is 
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taken  at  order  departure  from  the  factory,  regardless  if  they  make  it  to  the  customer  on 
time,  is  it  truly  meaningful  for  measuring  customer  satisfaction. 

Metrics  must  meet  the  organizations  needs  through  three  key  aspects  (Atkinson  et  al, 
1997;  Melnyk  et  al,  2004,  Fawcett  et  al,  2007).: 

1 .  They  must  correctly  drive  decision  behavior  and  provide  control 

2.  They  must  communicate  the  process  clearly  and  create  understanding 

3.  They  must  be  able  to  lead  to  improvement  within  the  process 

Metrics  exist  as  tools  to  be  used  by  the  appropriate  level  of  user  to  control  their  logistics 
process.  If  the  metric  provides  no  information  to  change  a  process  at  the  user  level,  it  is 
useless  and  worse  a  waste  of  resources  (Griffis,  2004).  The  best  metrics  communicate 
perfonnance  not  only  to  internal  sources  but  also  to  external  users.  Well-designed 
metrics  will  provide  users  with  information  on  a  process  without  needing  to  know  the 
nuts  and  bolts  of  the  operation  (Melnyk,  2004).  Finally  a  metric  must  lead  to  desired 
results.  If  a  company  uses  metrics  that  reflect  their  overall  strategic  direction,  then 
improvement  towards  those  goals  will  happen  (Fawcett  et  al.,2007).  Table  2.2 
establishes  a  checklist  to  evaluate  metrics  for  effectiveness. 
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Table  2.3.  Characteristics  of  an  Effective  Metric  (adapted  from  Fawcett  et  al,  2007) 


The  Metric  is.... 

Aligned  with  organizational  goals 

Customer  oriented 

Meaningful  to  workers,  managers,  and  customers 

Consistent  across  appropriate  functions  or  departments 

Promoting  cooperative  behavior  both  horizontally  and  vertically 

Communicated  to  all  relevant  individuals 

Simple,  Straightforward,  and  understandable 

Easy  to  collect  the  needed  data 

Easy  to  calculate 

Available  on  a  timely  basis 

Strategic  and  tactical 

Quantifiable 

Designed  to  drive  appropriate  behavior 

Designed  to  drive  learning  and  continuous  improvement 

Desinged  to  provide  information  that  is  actuallly  used  in  decision  making 

Ultimately  metrics  are  meant  to  indicate  how  a  process  is  working.  In  order  to  do  this, 
an  organization  must  establish  baseline  standards  of  perfonnance  measurement  that  align 
with  the  goals  of  the  organization.  By  establishing  this  standard,  metrics  can  easily 
indicate  gaps  between  performance  and  expectation  as  well  as  define  the  size  of  those 
gaps  (Melnyk,  2004;  Callahan,  2007;  Tsang,  1999). 


Air  Force  Aircraft  Maintenance  Metrics 

The  core  mission  of  the  air  force  is  to  defend  and  secure  the  free  use  of  the  air  and  space 
in  the  execution  of  national  objectives.  In  order  to  achieve  this  mission,  the  Air  Force 
must  fly  aircraft,  but  to  fly  aircraft  they  have  to  be  able  to  perform  maintenance. 
Maintenance  consists  of  all  the  activities  dedicated  to  maintaining  and  restoring  the 
physical  state  necessary  to  fulfill  its  production  mission  (Tsang,  1999).  The  U.S.  Air 
Force  has  established  a  comprehensive  framework  of  perfonnance  measures  in  order  to 
manage  its  maintenance  activities. 
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The  Metrics  Handbook  for  Maintenance  Leaders  (2001),  defines  the  two  cornerstones  of 
maintenance  metric  -  Aircraft  Fleet  Availability  and  Flying  Program  Execution.  Under 
these  two  primary  headings,  individual  metrics  are  categorized  as  leading  or  lagging 
indicators.  Leading  indicators  are  metrics  that  directly  impact  capability  to  provide 
resources,  while  lagging  indicators  show  firmly  established  trends  in  past  maintenance 
activities  (Air  Force,  2006). 

Aircraft  Fleet  Availability  is  measured  by  9  leading  indicators  and  10  lagging  indicators. 
Table  2.3  outlines  each  metrics  and  provides  both  the  definition  and  calculation  equation 
for  each  metric.  Leading  measures  are  Abort  Rate,  Code  3  Break  Rate,  Fix  Rate,  Repeat 
Rate,  Recur  Rate,  Logistics  Departure  Reliability,  Average  Deferred  discrepancies  per 
Aircraft,  Maintenance  Scheduling  Effectiveness  Rate,  Cannibalization  (CANN)  Rate,  and 
Phase/Isochronal  flow  rate.  Mission  Capable  Rate  with  its  derivative  sub-measures  Fully 
Mission  Capable  Rate,  Partially  Mission  Capable  Rate  -  Supply,  Maintenance  and  Both, 
Not  Mission  Capable  Rate  -  Supply,  Maintenance,  and  Both,  and  Total  Non  Mission 
Capable  Rate  -  Supply  and  Maintenance  make  up  the  list  of  lagging  indicators. 
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Table  2.4.  Aircraft  Fleet  Availability  Metrics  (Air  Force,  2006) 


Metric 

Definition 

Equation 

Leading  Indicators 

Abort  Rate 

Percentage  of  mission  that  end  prematurely 
and  must  be  re-accomplished 

(Air  +  Ground  Aborts/Total  Sorties  Flown  + 
Ground  Aborts)  x  1 00 

Break  Rate 

Percentage  of  aircraft  that  land  in  Code-3  or 
Alpha-3  (NMC)  status 

(Number  of  Sorties  that  land  Code-3/Total 
Sorties  Flown)  x  100 

Fix  Rate 

Percentage  of  aircraft  that  landed  Alpha-3  and 
returned  to  flyable  (FMC/PMC)  status  within 
set  time  window,  either  4,  8,  or  12  hours 

(Alpha-3  Breaks  fixed  within  window/Total 
Alpha-3  Breaks)  x  100 

Repeat  Rate 

Percentage  of  maintenace  discrepencies  that 
occur  again  after  next  sortie  attempt  after 
being  fixed 

(Total  Repeats/Total  Reported  Discrepencies) 
x  100 

Recur  Rate 

Percentage  of  maintenace  discrepencies  that 
occur  the  2nd  thru  4th  sortie  attempts  after 
being  fixed 

(Total  Recurs/Total  Reported  Discrepencies) 
x  100 

Logistics  Departure  Reliability  Rate 

Percentage  of  on-time  aircraft  departures  due 
to  logistics 

(Number  of  Departures-Number  of  Logistics 
Delays/Number  of  departures)  x  100 

Average  Delayed  Discrepency  Rate 

Average  Number  of  defered  maintenance 
actions 

(Total  (snapshot)  maintenace  actions/Average 
Aircraft  Possesed) 

Maintenace  Scheduling  Effectiveness  Rate 

Percentage  of  scheduled  maintenace  actions 
verses  accomplished  scheduled  maintenance 
actions  based  on  assigne  points  be  maintenace 
action 

(Total  points  earned/total  points  assigned)  x 
100 

CANN  Rate 

Percentage  of  cannibalization  actions  to 
replace  parts  on  other  aircraft 

(Number  of  CANNs/Total  Sorties  Flown)  x 
100 

Lagging  Indicators 

Mission  Capable  Rate 

Percentage  of  hours  aircraft  are  mission 
capable,  has  derivaties  of  Partial,  Non,  Total 
Non,  as  well  as  sub  catagories  of  Due  to 
Maintenace  or  Due  to  Supply 

(FMC  Hours  +  PMC  Hours/Possessed  hours) 
x  100 

Fully  Mission  Capable  Rate 

Percentage  of  hours  aircraft  can  perform  all 
assigned  missions 

(FMC  Flours/Total  Possessed  Flours)  x  100 

Partially  Mission  Capable  Rate 

Percentage  of  hours  aircraft  can  perform  some 
but  not  all  assigned  missions,  due  to  Supply, 
Maintenace  or  Both 

PMC  Flours/Total  Possessed  Flours)  x  100 

Non  Mission  Capable  Rate 

percentage  of  hours  aircraft  cannot  perform 
any  assigned  missions  due  to  Supply, 
Maintenace  or  Both 

(NMC  Hours/Total  Possessed  Hours)  x  100 

Total  Non  Mission  Capable  Rate 

Percentage  of  total  NMC  hours  -Supply, 
Maintenance  or  Both 

(total  NMC  Hours/Total  Possessed  Hours)  x 
100 

Flying  Program  Execution  is  measured  by  7  leading  and  2  lagging  indicators.  Primary 
Table  4  outlines  each  metrics  and  provides  both  the  definition  and  calculation  equation 
for  each  metric.  Aircraft  Inventory,  Possessed  Aircraft  Rate,  Programmed  Average 
Sortie  Duration  (ASD),  Actual  ASD,  Flying  Hour  Execution,  Flying  Scheduling 
Effectiveness  (FSE)  rate,  and  Chargeable  Deviation  Rate  are  all  leading  cause  indicators. 
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Lagging  indicators  of  program  execution  are  UTE  Rate  and  Logistics  Departure 
Reliability  Rate. 

Table  2.5.  Flying  Program  Execution  Metrics  (Air  Force,  2006) 


Metric 

Definition 

Equation 

Leading  Indicators 

Flying  Hour  Execution 
Rate 

Percentage  of  Actual  Flying  hours 
executed  verses  planned  flying  hours 

(Total  Flying  hours  executed/Total 
Flying  Hours  Scheduled)  x  100 

Flying-scheduling 
Effectiveness  Rate 

Percentage  of  deviations  from 
planned  flying  schedule  to  actual 
flying  schedule 

(Actual  Flying  Schedule  - 
Deviations/ Actual  Flying  Schedule) 
x  100 

Chargable  Deviation 
Rate 

Percentage  of  Deviations  due  to 
Maintenace  or  Operations  actions 

(Total  Mx  deviations+total  Ops 
deviations/Total  sorties  Scheduled) 
x  100 

Primary  Aircraft 
Inventory 

Assigned  number  of  aircraft  by  MDS 
per  designated  organization 

Snapshot  of  number  of  aircraft 
assigned 

Possessed  Aircraft  Rate 

Aircraft  under  control  of  owning 
designated  organization 

Snapshot  of  number  of  aircraft 
controled  by  organization 

Programed  Average 
Sortie  Duration 

Average  Sorte  length  scheduled 

(Number  of  Sorties/Total  scheduled 
Sortie  Hours) 

Actual  Average  Sortie 
Duration 

Average  Sortie  length  executed 

(Number  of  Sorties  executed/Total 
executed  Sortie  Hours 

Lagging 

Logistics  Departure 
Reliablity  Rate 

Percentage  of  on-time  aircraft 
departures  due  to  logistics 

((#  of  Departures  -  #  Logistics 
delays)/#  of  Departures  x  100) 

Utilization  Rate 

Average  number  of  Sorties  flown  per 
Primary  Aircraft  Inventory 

(Sorties/Hours  flown  per 
month/P AI  per  Month) 

These  represent  the  core  metrics  utilized  by  most  aircraft  maintenance  mangers  to  run 
their  maintenance  management  processes.  They  are  the  ones  most  discussed  and  talked 
about  from  the  flight  line  to  the  Pentagon.  Based  on  these  observations  these  metrics 
were  selected  by  this  study  for  evaluation  and  analysis. 
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Summary 

The  impetus  for  change  in  operating  processes  to  any  organization  can  come  from  many 
sources.  New  customer  requirements,  changes  in  corporate  policy,  increasing  global 
competition,  and  shrinking  operational  budgets  are  driving  the  need  to  change  in  the 
business  world.  The  next  evolution  in  logistics  management  created  to  meet  this  need  for 
change  is  Enterprise  Resource  Planning.  Organizations  are  spending  billions  of  dollars 
on  establishing  new  programs  in  order  to  fully  integrate  all  aspects  of  their  operations  in 
order  to  realize  the  fruits  of  system  wide  optimization.  The  US  Air  Force  is  not  immune 
to  these  same  change  characteristics  as  it  is  seeking  the  similar  operational  changes 
through  ECSS. 

This  literature  review  has  outlined  the  drive  for  change  in  the  Department  of  Defense 
logistics  operations  and  its  effect  on  future  US  Air  Force  logistical  programming  and 
planning.  The  review  then  discussed  the  Critical  Success  Factor  for  successful  ERP 
implementation  as  proposed  by  the  academic  and  operational  community.  The  final 
sections  discussed  the  characteristics  of  effective  logistics  process  measurements  and 
reviewed  the  Air  Force’s  Aircraft  Maintenance  metrics  program  that  will  need  to  be 
converted  into  the  proposed  ERP  implementation  program.  Effective  integration  of  the 
legacy  aircraft  metrics  system  will  allow  true  process  oversight  and  control  across  the 
maintenance  enterprise,  providing  one  step  in  achieving  true  logistics  collaboration  and 
control  envisioned  by  the  Department  of  the  Air  Force  and  the  Department  of  Defence. 
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III.  Methodology 


Overview 

All  journeys  have  a  goal  and  in  order  to  reach  that  goal  they  must  begin  with  a  plan. 
Research  is  no  different,  it  is  merely  a  journey  of  the  mind,  and  so  it  also  requires  a  map. 
This  chapter  outlines  the  path  this  study  will  use  to  accomplish  this  journey  of  discovery. 
The  goal  of  this  study  is  to  develop  a  comparison  framework  for  translating  current 
maintenance  metrics  into  the  new  ECSS  system  and  to  validate  this  framework  by 
conducting  field  interviews  of  aircraft  maintenance  managers.  The  first  section  in  this 
chapter,  the  methodology  paradigm  chosen  to  frame  the  research  has  been  outlined  and 
discussed.  This  study  has  chosen  to  use  a  Qualitative  Analysis  based  Multiple  Case 
approach  as  the  most  suitable  for  the  problem  posed.  Second,  the  chapter  details  the 
research  questions  and  lays  out  the  research  design,  to  include  the  development  of  the 
comparison  framework,  thereby  setting  the  roadmap  to  accomplish  the  goal  of  this  study. 
As  the  study  seeks  to  validate  the  framework  developed,  a  structured  interview  guide  for 
use  by  individual  subject  matter  experts  to  evaluate  the  selected  metric  cases  was 
developed.  Finally,  data  collection  methods,  analysis  of  the  collected  data,  and  validity 
and  reliability  requirements  that  support  the  study’s  conclusions  have  been  addressed. 

Methodology 

Methodology,  as  defined  by  Strauss  and  Corbin  (1998),  is  a  way  of  thinking  about  and 
studying  reality,  and  will  gradually  move  us  towards  a  greater  understanding  of  the 
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world.  Qualitative  Field  Research  is  one  of  the  many  defined  methods  of  researching 
reality.  Qualitative  research  provides  insight  into  questions  involving  the  “how”  and 
“why”  of  a  subject  of  interest  (Yin,  2009).  Typically,  data  collected  does  not  result  in 
data  that  can  be  used  in  statistical  or  quantitative  analysis  (Babbie,  2005).  While  this 
does  not  preclude  quantitative  analysis  of  qualitative  data,  the  primary  focus  should  be  on 
uncovering  and  understanding  the  underlying  patterns  and  structures  revealed,  not  the 
numerical  results  (Babbie,  2005). 

Under  the  qualitative  field  research  method,  there  exist  several  approaches  that  can  be 
used  to  accomplish  the  researcher’s  goals.  This  study  was  based  on  the  Grounded  Theory 
Approach  accomplishing  a  multiple  case  study  analysis  of  the  developed  comparison 
framework.  By  using  the  underlying  principles  of  Grounded  Theory,  units  of  analysis 
and  subject  matter  experts  can  be  selected  based  on  their  relevance  to  evaluate  the 
proposed  framework  under  question. 

Research  Questions 

While  research  usually  begins  with  some  problem  being  defined,  in  order  to  achieve  the 
systematic  structure  that  is  research,  a  research  question  whose  answer  provides  a 
solution  to  the  problem  is  needed  (Booth  et  al,  2008).  Therefore,  this  study  has  proposed 
the  following  questions: 

1 .  How  well  do  current  Air  Force  maintenance  metrics  translate  into  the  proposed 
Expeditionary  Combat  Support  System  Enterprise  Resource  Planning  framework? 
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2.  Which  current  Air  Force  maintenance  metrics  should  be  translated  into  the 


proposed  Expeditionary  Combat  Support  System  Enterprise  Resource  Planning 
framework? 

These  questions  provide  the  overall  framework  to  structure  this  study  in  order  to  provide 
solutions  to  the  initial  problem  statement  from  Chapter  1 .  However,  in  order  to  answer 
the  questions,  several  investigative  questions  were  developed  to  further  refine  the 
direction  of  the  study. 

Investigative  Questions 

The  following  questions  were  derived  directly  from  the  initial  research  questions. 

1 .  How  important  is  the  current  metric  for  managing  aircraft  maintenance? 

2.  How  effective  is  the  current  metric  in  managing  aircraft  maintenance? 

3.  How  adequately  does  the  proposed  translated  metric  replace  the  current  metric? 

4.  How  easily  understood  is  the  new  proposed  metric? 

These  questions  further  focused  the  direction  of  the  research  and  provided  a  basis  to 
accomplish  both  the  proposed  framework  construction  and  to  fully  leverage  the  subject 
matter  experts  in  the  evaluation  of  the  framework. 

Unit  of  Analysis 

Babbie  states  “In  social  research,  there  is  virtually  no  limit  to  what  or  whom  can  be 
studied,  or  units  of  analysis.”  Fortunately,  for  most  research  the  units  of  analysis  are 
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fairly  obvious  and  intrinsic  to  the  research  question  (Babbie,  2005).  In  an  organization  as 
large  as  the  Air  Force,  there  exist  thousand  of  management  programs  with  tens  of 
thousands  of  potential  metrics,  Figure  3.1.  In  order  to  conduct  a  realistic  study  a  small 
portion  of  this  organization  will  be  looked  at.  For  this  study  the  units  of  analysis  selected 
are  two  maintenance  metrics  sub  programs,  Aircraft  Fleet  Availability  and  Flying 
Program  Execution. 


Tactical 


Air  Force  Maintenance  Metrics  Programs 


Figure  3.1.  Unit  of  Analysis  Selection 


Research  Design 

As  stated  previously,  the  strategy  of  this  study  is  to  develop  a  comparative  framework  for 
incorporating  current  metrics  into  the  new  ECSS  system  and  to  validate  this  comparison 
by  leveraging  aircraft  maintenance  subject  matter  expert  field  interviews.  This  goal 
defines  two  areas  of  execution;  the  first  part  involved  the  development  of  the  comparison 
framework  to  translate  the  existing  system  of  measurement  to  the  future  state  system. 

This  process  will  leverage  the  experiences  of  the  ECSS  development  team  to  create  a 
clear  and  complete  matrix  to  provide  to  the  selected  experts  for  evaluation.  The  second 
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part  is  to  conduct  qualitative  field  interviews  of  the  experts  to  collect  data  on  their 
evaluation  of  the  proposed  metrics.  The  results  of  this  field  research  will  substantiate  and 
validate  the  framework  and  provide  recommendation  for  disposition  of  the  metrics. 

Metrics  Comparison  Framework 

The  twenty  eight  metrics  identified  in  Chapter  2  (see  Tables  2.3  and  2.4),  were  selected 
for  translation  by  this  study.  In  order  to  translate  these  metrics  clearly,  a  three  step 
process  was  developed.  Each  metric  was  analyzed  for  key  attributes,  and  then  compared 
to  the  perfonnance  measures  provided  by  the  Logistics  Transfonnation  Office  based  on 
the  SCOR  model.  Finally  each  SCOR  metric  is  compared  with  the  Key  Perfonnance 
Indicators  identified  by  the  Oracle  software  suite  operation  manuals. 

First  each  metric  was  identified  by  three  key  attributes: 

1 .  Nomenclature 

2.  Definition 

3.  Equation 

These  three  attributes  provide  the  core  identity  for  each  metric  and  existed  across  all  three 
models.  Comparative  pattern  analysis  was  then  used  to  evaluate  each  metric  against  the 
list  of  potential  SCOR  model  metrics  in  Appendix  B.  Once  these  attribute  patterns  were 
compared  across  all  selected  metrics,  each  was  categorized  based  on  number  of 
similarities  of  these  key  attributes,  either  level  3,  level  2,  level  1,  or  no  similarity.  Level 
3  pairings  matched  on  all  three  key  attributes,  Level  2  across  two  attributes,  Level  1 
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matched  on  only  one  key  attribute  category,  and  No  Connection  had  no  matching 
attributes.  This  process  was  then  repeated  comparing  the  attributes  between  the  SCOR 
model  metrics  and  the  Key  Perfonnance  Indicators  built  into  the  Oracle  software 
platform.  The  results  of  the  comparative  analysis  and  the  metric  evaluations  are 
discussed  in  Chapter  4. 


Once  the  comparison  framework  was  developed  for  all  28  metrics,  it  was  then 
incorporated  into  the  interview  guide.  Each  of  the  four  investigative  questions  was 
evaluated  on  a  5  point  Likert  scale  for  each  metric  comparison.  Figure  3.2  shows  a 
sample  of  one  interview  question  set,  the  full  interview  guide  can  be  found  in 


Appendix  A. 


Metric  24 

Current  Proposed 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


Utilization  Rate 


A  Hcisirc  of  the  average  kc  of  a 
systea  dung  a  specified  period  of 
cakadartme 
Average  asage/tiae  -  flyiag 
fcrs/PAA  per  aoilk  (assigned) 


Utilization  Rate 


Average  ■■■ber  of  Sorties/Hoars 
Dowa  per  Priaary  Aircraft 
laveatory 

(Sorties/Hoars  flows  per 
aoatfc/PAI  per  Moat  h) 


Figure  3.2  Sample  Interview  Guide  Question  Set 
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Data  Source  Selection 


As  with  all  other  areas  of  qualitative  research,  selection  of  the  sources  of  data  must  be 
systematic  and  purposeful.  Selection  of  the  wrong  characteristics  can  lead  to  faulty 
conclusions  or  invalid  findings,  thus  the  following  criteria  were  used  in  SME  selection: 

1 .  The  SME  should  be  or  have  been  an  aircraft  maintenance  manager  for  7  years. 

2.  The  SME  should  have  experience  with  maintenance  metrics. 

3.  The  SME  must  be  available  to  be  interviewed  by  the  researchers. 

The  most  important  of  these  criteria  is  SME  availability,  if  the  subject  is  not  available  for 
interview  than  this  research  cannot  be  completed.  Following  this,  experience  and 
familiarity  with  the  subject  are  critical.  While  this  study  proposes  new  translations  of 
current  performance  measurements,  the  fundamental  principles  of  aircraft  maintenance 
remain  the  same.  The  SME  collective  experience  in  the  aircraft  maintenance  enterprise 
should  allow  them  to  effectively  evaluate  the  metrics  and  their  proposed  translations. 

Data  Collection 

The  primary  source  of  data  collection  was  qualitative  interviews  with  experts  in  the  field. 
The  study  collected  data  from  one  population,  Senior  Maintenance  Managers.  12  Subject 
Matter  Experts  were  selected  from  Charleston  AFB  due  to  the  ease  of  access  to  the 
location,  time  constraints  and  available  personnel  schedules.  Senior  managers  were 
selected  in  order  to  capture  a  mix  of  operational  experience.  Air  Force  operations 
encompass  a  diverse  range  of  aircraft  mission  functions,  from  Fighter  to  Strategic  Airlift. 
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Senior  managers  will  have  a  greater  range  of  experience  across  these  diverse  mission 
environments,  verses  less  experienced  managers  who  may  only  understand  one  mission 
type.  The  total  cumulative  experience  of  the  SMEs  was  211  years,  at  an  average  of  16.3. 
While  the  predominate  experience  was  in  Airlift  -  5 1%,  there  was  21%  with  Fighter 
experience,  9%  with  Staff  Experience,  6%  in  Special  Operations  with  the  remainder  in 
Tanker,  Command  and  Control,  Acquisitions  and  Trainer  aircraft.  Figure  3.1  shows  the 
breakdown  of  respondent  experience  by  type  category. 


4%1%2%  9% 

■  Airlift 

■  Figjkter 

■  Tanker 

i ]  Command  and  Control 

■  Special  Operations 

■  Acquisitions 

■  Space  Assets 
Trainer  Aircraft 


Figure  3.3  Respondent  Experience  Type  by  Category 

The  experts  were  selected  via  requests  to  their  respective  commanders  (Appendix  D). 
Then  individual  requests  were  sent  to  perspective  managers  (Appendix  E).  Once  an 
agreement  to  be  interviewed  was  reached,  interview  times  were  established  and  the 
interviewees  were  provided  the  translation  framework  prior  to  the  interview.  The 
interviews  were  conducted  from  November  2008  to  January  2009  and  recorded  for  ease 
of  data  capture.  After  all  interviews  were  completed,  the  researchers  transcribed  and  then 
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summarized  each  interview.  Each  transcript  was  sent  to  the  respective  interviewee  to 
verify  the  areas  discussed  and  ensure  their  words  were  captured  accurately. 

Data  Analysis 

All  researchers,  just  as  artists  and  engineers,  need  tools  to  help  them  accomplish  their 
goals  (Strauss  and  Corbin,  2008).  The  tools  of  the  artist  are  the  brush  and  palette,  while 
the  engineer  will  use  the  calculator  and  T-square.  Just  as  the  tools  to  the  trades  are  many 
and  varied,  so  are  the  analytical  tools  of  the  researcher.  Analyses  by  questioning, 
analysis  of  words,  or  even  via  comparisons  are  just  a  few  of  the  tools  available  to 
comprehend  the  huge  amount  of  data  that  can  be  collected  in  qualitative  research. 

This  study  has  chosen  to  apply  a  systematic  comparison  of  two  or  more  phenomena,  in 
order  to  quantify  and  categorize  the  provided  data  (Strauss  and  Corbin,  2008).  The  study 
developed  a  coding  sequence  in  order  to  organize  the  field  data  results.  Each  SME  was 
provided  the  metrics  framework  and  then  interviewed  on  their  evaluation  of  the  new 
metrics.  The  interview  data  results  were  then  consolidated  and  evaluated  for  common 
underlying  theme  or  patterns  of  positive  or  negative  evaluations.  Further  detailed 
analysis  and  the  results  of  the  data  findings  can  be  found  in  Chapter  4. 

Validity  and  Reliability 

Every  study  seeks  two  underlying  goals  when  accomplishing  their  research,  establish 
validity  and  reliability.  Four  tests  have  been  identified  as  common  to  all  social  science 
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research;  construct  validity,  internal  validity,  external  validity,  and  reliability  (Yin,  2007). 
Each  of  these  tests  has  its  own  techniques  and  tactics  for  maximizing  their  presence  in 
any  research.  Table  3.1  outlines  the  four  tests  and  how  this  study  will  address  each  one. 

Table  3.1,  Validity  and  Reliability  Table  (adapted  from  Yin,  2009) 


Test 

Tactic 

Phase  of  tactic 

Construct  Validity 

Multiple  sources  of  evidence 

Data  Collection 

Chain  of  eveidence 

Data  Collection 

Key  informants  review  draft 

interview  summaries 

Compsition/analysis 

Internal  Validity 

Pattern  matching 

Analysis 

Explanation  building 

Analysis 

Logic  models 

Analysis 

External  Validity 

Replication  logic 

Research  design 

Reliability 

Use  case  study  protocal 

Research  design 

Construct  validity  was  achieved  for  this  study  by  selecting  12  maintenance  management 
experts  to  evaluate  the  proposed  framework,  maintaining  the  linkages  from  respondents 
data  through  to  the  summary  analysis,  and  each  respondent  reviewed  and  approved  their 
interview  transcript  summaries.  Internal  validity  was  maintained  in  the  development  of 
the  framework  through  use  of  pattern  matching  of  the  three  key  attributes  for  comparison 
of  the  metrics.  Additional  development  of  the  coding  analysis  framework  in  Appendix  C 
allowed  for  pattern  matching  analysis  of  the  interview  data.  External  validity  is 
established  though  repeated  application  of  translation  framework  and  interview  guide  and 
comparing  results  to  this  initial  pilot  exploration. 


Reliability  in  qualitative  studies  has  been  the  harder  of  the  two  properties  to  establish. 

Reliability  in  any  qualitative  study  can  be  heavily  influenced  by  researcher  bias.  This 

study  has  attempted  to  achieve  reliability  by  two  means.  First,  the  translation  framework 
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has  been  built  through  close  evaluation  of  metric  attributes  and  clearly  documented. 
Second,  the  field  interview  guide  was  developed  based  on  the  investigative  questions. 

The  guide  was  built  for  each  metric  using  a  standard  format  that  may  be  easily  replicated 
for  the  metrics  studied  or  on  any  other  metrics  of  interest. 

Summary 

This  chapter  has  outlined  the  plan  used  to  accomplish  the  strategy  of  this  study  - 
developing  a  framework  for  translating  metrics  into  the  new  ECSS  system  and  validating 
this  comparison  by  conducting  field  interviews  aircraft  maintenance  experts.  All  the 
necessary  steps  to  achieve  satisfactory  results  have  been  outlined  above.  The  initial 
problem  statement  and  subsequent  research  questions  have  established  the  need  for  a 
qualitative  research  paradigm.  The  grounded  theory  approach  was  used  to  develop  the 
translation  framework  and  validate  it  through  systematic  interviews  to  generate  an 
solution  to  answer  the  research  questions.  Finally  the  discussion  of  data  source  collection 
and  analysis  has  defined  the  necessary  steps  that  will  lead  to  establishing  validity  and 
reliability  within  the  study. 
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IV.  Analysis 


Overview 

“If  you  can’t  measure  it,  you  can’t  manage  it”.  This  is  the  classic  saying  that  captures  the 
essential  reality  of  the  maintenance  manager.  Good  performance  measurement  is  critical 
to  the  effective  control  of  any  logistics  process.  It  is  important  to  ensure  current  process 
metrics  are  reviewed  and  evaluated  before  incorporating  them  into  a  new  management 
system.  This  chapter  will  outline  the  empirical  results  of  the  comparison  and  evaluations 
of  the  28  aircraft  maintenance  metrics  selected  for  this  thesis.  The  chapter  will  first 
provide  the  analysis  of  each  metric  case.  Then  cross  case  analysis  of  the  metric 
characteristics  derived  from  the  interviews  will  be  presented.  Finally  the  significant 
findings  of  the  research  questions  will  be  discussed. 

Metric  Case  Evaluation 

Each  metric  was  evaluated  by  aircraft  maintenance  managers  on  a  5  point  Likert  scale  for 
each  investigative  question.  Questions  1  and  2  evaluated  the  importance  and 
effectiveness  of  the  metric,  while  questions  3  and  4  evaluated  the  clarity  of  the  translation 
and  understanding  of  the  metric.  Evaluation  scores  had  a  possible  range  of  2  points  to  10 
points  for  individual  ratings,  and  a  range  of  24  points  to  120  points  possible  for  the 
aggregate  scores.  Metric  evaluation  score  and  interview  findings  were  used  to  categorize 
each  metric  as  high,  moderate,  or  low  for  importance  and  effectiveness  and  clear, 
moderate  or  unclear  for  translation  and  understanding.  Aggregate  scores  for  high  rated 
metrics  were  between  90  to  120,  Moderate  metric  scores  were  between  80  and  89,  and 
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Low  rated  scores  were  79  and  below.  Aggregate  scores  for  Clear  evaluations  were 
between  100  and  120,  Moderate  between  80  to  99,  and  Unclear  metrics  were  scored 
below  79.  Metrics  with  no  comparisons  did  not  receive  translation  and  understanding 
evaluations,  hence  no  scores.  Summary  tables  of  all  metric  evaluations  can  be  found  in 
Tables  4.5  and  4.6. 

Abort  rate 

Comparison 

Abort  rate  as  defined  by  the  literature  is  the  percentage  of  missions  that  end  prematurely 
and  must  be  re-accomplished  (Air  Force,  2006).  This  metric  is  calculated  by  the 
summation  of  Air  and  Ground  aborts  divided  by  Total  sorties  flown  plus  Ground  aborts. 
After  comparing  all  three  key  attributes  no  metric  was  found  to  be  match  in  either  the 
SCOR  model  or  Oracle  system.  This  metric  was  only  evaluated  for  importance  and 
effectives  in  the  current  system. 


Current  Metric 

X 

SCOR  Metric 

Oracle  Metric 

Air  Aborts  +  Ground  Aborts 

Total  Sorties  +  Ground  Aborts 

N- Nomenclature 

N- Nomenclature 

N- Definition 

N- Definition 

N- Equation  N- Equation 


Investigative  Question  results 

Importance  Effectiveness 

Translation  Understanding 

Low 

N/A 

Figure  4.1  Abort  Rate  Metric  Summary 
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Importance  and  Effectiveness 

This  metric  consistently  rated  low  in  importance  and  effectiveness  across  most 
interviewees.  Data  captured  by  this  metric  was  noted  to  also  be  captured  by  other 
metrics,  such  as  Break  rate  under  Code/Alpha-4  landing  status.  Respondents  also 
identified  that  this  metric  was  not  used  for  any  management  decision,  it  was  tracked  and 
reported  but  no  actions  were  taken  in  response  to  this  metric.  Abort  rate  seems  to  have 
aircraft  system,  or  Mission  Design  Series  (MDS),  specific  impacts.  For  example,  on 
older  systems  such  as  the  C-5  or  F-15  this  metric  was  important  due  to  higher  incidents  of 
abort  events,  but  in  newer  systems  such  as  the  C-17  it  had  so  few  events  that  Abort  rate 
was  identified  as  being  meaningless.  One  respondent  felt  that  abort  rate  was  both 
important  and  effective  but  did  not  provide  any  reasoning  into  why  they  marked  it  high. 
Two  respondents  stated  while  they  think  the  metric  is  not  important,  it  does  play  a  role  in 
mission  forecasting.  With  an  increased  abort  rate,  more  missions  will  have  to  be 
generated,  this  in  turn  means  more  airframes  will  have  to  be  scheduled  to  fill  these  lines. 

Translation  and  Understanding 

After  reviewing  the  SCOR  metrics  list  and  Oracle  software  manuals  provided  by  the 
Logistics  Transfonnation  Office  (Appendix  B),  the  study  found  that  there  is  no  metric  to 
correlate  with  abort  rate.  Therefore  this  metric  was  evaluated  only  on  Importance  and 
Effectiveness  investigative  questions. 
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Break  rate 


Comparison 

Break  rate  is  the  percentage  of  airframes  that  land  broken  or  in  landing  status 
Code/Alpha-3  (Air  Force,  2006).  When  aircraft  land  they  are  given  landing  status  codes 
based  on  what  maintenance  is  required  for  the  aircraft.  Aircraft  are  given  status 
Code/Alpha-1  for  fully  mission  capable,  status  Code/Alpha-2  for  partially  maintenance 
capable  and  status  Code/Alpha-3  for  non  mission  capable.  This  metric  is  calculated  by 
taking  the  number  of  sorties  that  arrive  in  Code/Aplha-3  divided  by  total  sorties  flown. 
By  comparing  the  3  key  attributes  identified  in  Chapter  3,  to  the  metrics  available  in  the 
SCOR  database  (Appendix  B),  one  comparison  was  identified,  Break  rate  (Metric  729). 
Break  rate  matched  the  current  metric  on  all  three  key  attributes,  Nomenclature, 
Definition,  and  Equation;  making  it  a  Level  3  comparison.  No  metrics  were  found  in  the 
Oracle  software  similar  to  break  rate.  This  trend  was  found  throughout  most  of  the 
metrics.  The  Break  Rate  set  was  then  evaluated  by  all  4  investigative  questions. 
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Figure  4.2  Break  Rate  Metric  Summary 
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Importance  and  Effectiveness 

Break  rate  appeared  to  have  mixed  reaction  among  the  interview  ratings  for 
importance/effectiveness.  Respondents  identified  several  factors  that  make  break  rate  an 
unimportant/in-effective  metric.  They  stated  that  the  metric  may  be  important  at  a 
strategic  staff  level  because  it  could  lead  to  decisions  on  how  to  budget  and  allocate  for 
parts  and  equipment.  On  a  tactical  decision  level  one  respondent  rated  it  not  important 
and  not  effective  because  “things  break  and  you  can’t  control  when  things  break”. 
Additionally  this  metric  was  found  to  be  susceptible  to  manipulation  through  intentional 
acts  or  personnel  misunderstanding  how  to  capture  this  data.  When  an  aircraft  lands  the 
crew  is  debriefed  on  its  maintenance  performance,  if  an  maintainer  does  not  correctly 
status  the  aircraft  Code/Alpha-3  then  the  data  may  be  skewed.  Several  respondents  rated 
the  metric  important  and  effective  due  to  the  metrics  ability  to  easily  trends  captured 
maintenance  information  and  provide  detailed  histories  by  aircraft.  Additionally  the 
metric  is  important  to  MDS  scheduling  because  planes  with  certain  maintenance  histories 
are  kept  from  important  and  high  profile  missions.  This  metric  was  also  found  to  be  a 
barometer  of  maintenance  response  to  measure  how  effective  personnel  were  at 
responding  to  and  correcting  maintenance  issues. 

Translation  and  Understanding 

Break  rate  did  measure  consistently  clear  in  translation/understanding  across  all 
interviews.  Respondents  found  the  proposed  change  from  the  old  to  the  new  metric 
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sufficient  and  understandable.  By  being  able  to  maintain  the  same  nomenclature, 
definitions  and  equation,  they  felt  it  maintained  the  essence  of  the  original  metric. 

Fix  rate 

Comparison 

Fix  rate  is  defined  as  the  percentage  of  aircraft  that  landed  in  status  Code/Alpha-3  and 
returned  to  flyable  status  within  a  set  time  range,  either  4,  8,  or  12  hours  (Air  Force, 
2006).  When  aircraft  land  they  are  given  landing  status  codes  based  on  what 
maintenance  is  required  for  the  aircraft.  Fix  rate  is  calculated  by  taking  the  number  of 
code  3  breaks  fixed  within  the  defined  period  divided  by  total  code-3  breaks.  In 
comparing  the  key  attributes  to  the  SCOR  list  (Appendix  B),  the  study  found  Fix  Rate 
(Metric  732)  matched  on  all  three  criteria.  Nomenclature,  definition,  and  equations  all 
matched  exactly;  therefore  Fix  rate  merits  a  Level  3  comparison. 
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Figure  4.3  Fix  Rate  Metric  Summary 

Importance  and  Effectiveness 

Fix  rate  scored  consistently  high  across  respondents  for  importance/effectiveness.  This 
metric  seems  to  have  the  strongest  impact  on  management’s  ability  to  utilize  personnel 
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capabilities.  Fix  rate  can  tell  you  where  training  deficiencies  lie,  as  well  as  where  you 
have  skill  assignment  imbalances  either  shift  to  shift  or  unit  to  unit.  Additionally 
respondents  spoke  to  how  fix  rate  can  allow  production  managers  to  determine 
maintenance  repair  priorities,  a  critical  need  in  maintenance  planning.  This  metric  also 
seems  to  be  aircraft  specific  in  how  important  it  was  rated.  Maintenance  managers  of 
older  airframes  seem  to  rate  a  higher  level  of  attention  to  fix  rate  than  managers  of  newer 
systems.  These  results  are  probably  due  to  the  nature  of  older  aircraft.  They  tend  to 
break  more  frequently  and  repairs  are  more  difficult.  In  spite  of  the  overall  importance 
placed  on  this  metric,  two  respondents  rated  this  metric  low  because  they  don’t  use  it  to 
make  management  decisions.  There  was  a  clear  understanding  of  the  metric  but  lack  of 
senior  level  attention  deemed  it  unimportant. 

Translation  and  Understanding 

As  found  in  the  previous  metric,  fix  rate  is  rated  a  level  3  comparison,  this  seems  to 
contribute  to  a  clear  rating  of  translation/understanding  across  all  interviews. 
Respondents  found  the  proposed  change  from  the  old  to  the  new  metric  sufficient  and 
understandable.  By  being  able  to  maintain  the  same  nomenclature,  definitions  and 
equation,  they  felt  it  maintained  the  essence  of  the  original  metric. 
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Repeat  rate  and  Recur  rate 

Comparison 

Repeat  rate  is  the  percentage  of  maintenance  discrepancies  that  occur  again  during  the 
next  sortie  attempt  after  being  fixed  (Air  Force,  2006).  Recur  rate  is  the  percentage  of 
maintenance  discrepancies  that  occur  on  the  second  thru  fourth  sorties  after  being  fixed. 
Both  metrics  use  a  similar  equation  for  calculation.  Total  repeats  (Total  Recurs)  divided 
by  total  reported  discrepancies.  After  using  the  key  attributes  from  both  equations,  no 
metric  was  found  in  the  SCOR  list  that  corresponded  to  either  repeat  rate  or  recur  rate. 
However  upon  review  of  the  Oracle  key  performance  indicator  listing  (Oracle,  2007),  the 
metric  First  Time  Fix  rate  was  selected  as  a  suitable  candidate  for  conversion.  This 
metric  did  not  correlate  on  nomenclature  with  repeat  and  recur;  but  the  study  found  that 
the  definitions  had  an  inverse  relationship.  While  Repeat/Recurs  captured  data  based  on 
failed  fixes,  First  Time  Fix  Rate  captured  the  data  based  on  successful  fixes.  First  time 
fix  rate  is  calculated  by  taking  the  total  first  time  fixes  divided  by  total  maintenance 
requests,  where  requests  were  equal  to  discrepancies.  The  equations  were  equivalent  if 
one  takes  into  account  the  inverse  relationship  between  the  two  metrics.  Therefore  this 
metric  was  categorized  as  a  Level  2  comparison.  An  added  bonus  of  the  inverse 
relationship  is  that  both  current  metrics  can  be  combined  in  the  translation  allowing  for 
fewer  metrics  for  the  user  to  have  to  track  and  manage. 
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Figure  4.4  Repeat  and  Recur  Rate  Metric  Summary 

Importance  and  Effectiveness 

Repeat/Recur  was  again  consistently  rated  high  across  all  interviews  for  importance  and 
effectiveness  of  both  current  metrics.  This  metric  was  found  to  contribute  directly  to 
Manning  utilization  management.  Respondents  identified  that  this  metric  will  tell 
managers  how  well  the  maintenance  personnel  are  accomplishing  their  assigned  task.  It 
will  help  flag  critical  maintenance  issues  and  help  manager’s  determine  if  there  are  skill 
proficiency,  equipment,  or  technical  data  problems.  Repeat/Recur  rates  impact 
operational  readiness  through  its  ability  to  identify  maintenance  issues  that  may  be 
systemic  across  the  fleet.  While  all  interviews  rated  repeat/recur  rates  as  important, 
several  also  identified  potential  downsides  to  their  effective  implementation.  One  of  the 
most  frequently  mentioned  was  its  relevance  across  different  MDS’s.  Fighter  aircraft  are 
fairly  static  in  terms  of  off  station  verses  home  station  operations,  thus  management 
systems  are  more  likely  to  capture  all  repeat  events.  However  in  the  airlift  world, 
airframes  often  spend  more  time  off  station  than  at  home,  sometimes  over  45  days  at  a 
time,  and  usually  never  more  than  1-2  days  in  one  location.  This  creates  a  problem  for 
capturing  repeat  maintenance  events. 
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Translation  and  Understanding 

Respondent’s  ratings  of  First  Time  Fix  Rate  were  consistent  across  most  interviews; 
however  the  metric  was  rated  moderate  for  translation/understandability.  Most  of  those 
surveyed  required  a  second  look  to  see  how  the  metrics  changed  from  Repeat/Recur  to 
First  Time  Fix  Rate.  The  fact  that  they  have  an  inverse  relationship  was  identified  as  the 
primary  reason  for  marking  it  down  in  this  category.  Once  they  had  time  to  assimilate 
the  new  metric  it  was  found  to  be  favored  over  the  current  metrics  as  a  tool  for 
maintenance  management. 

Logistics  Departure  Reliability  Rate 

Comparison 

Logistics  Departure  Reliability  (LDR)  or  sometimes  known  as  Departure  Reliability  is 
percentage  of  on  time  aircraft  departures  due  to  logistics  (Air  Force,  2006).  Departure 
Reliability  is  calculated  by  taking  the  number  of  aircraft  departures  minus  number  of 
logistics  delays  divided  by  the  number  of  departures.  Based  on  the  key  attribute  coding, 
On  time  Operation  Starts  (Metric  744)  in  the  SCOR  list  was  similar  to  Departure 
Reliability  along  two  attributes:  metric  definition  and  equation  (Appendix  B).  On  Time 
Operation  Starts  is  defined  as  the  percentage  of  operations  started  on  time,  and  calculates 
it  by  dividing  the  number  of  operation  on  time  starts  by  the  number  of  operations.  By 
equating  aircraft  departures  to  operation  starts,  this  study  has  found  they  are  equivalent 
metrics  along  two  aspects,  and  are  consequently  categorized  as  a  Level  2  comparison. 
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Figure  4.5  LDR  Rate  Metric  Summary 

Importance  and  Effectiveness 

Departure  Reliability  was  rated  consistently  as  moderately  important/effective  across  the 
interviews,  with  only  two  rating  it  moderately  low  in  importance  and  effectiveness. 
While  Departure  Reliability  was  found  to  be  able  to  identify  mission  execution  issues, 
respondents  stated  it  fails  to  capture  the  total  picture  of  an  aircraft  launch  operation.  The 
metric  would  be  more  effective  if  it  were  to  capture  all  the  launch  events  within  the 
established  schedule  of  events.  In  doing  this  specific  process  breakdowns  could  be 
highlighted  and  appropriate  management  fixes  implemented.  Ultimately  policy 
decisions  would  dictate  how  effective  this  metric  could  be.  One  potential  alternative  to 
Departure  Reliability  rate  offered  was  On  Time  Operation  Completion.  Respondents 
indicated  that  the  more  important  issue  is  not  if  an  aircraft  launched  on  time,  but  did  it 
deliver  its  intended  mission  when  required.  As  one  manager  put  it  “did  we  get  it  to  the 
customer  when  they  asked  for  it?  This  is  applicable  whether  it  was  a  tank,  bomb,  aerial 
refueling  or  surveillance  asset.” 
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Translation  and  Understanding 

Translation/understanding  was  rated  moderate  for  this  metric.  Determining  what 
constitutes  operation  starts  was  a  key  item  identified  that  negatively  affected  the 
understanding  of  this  metric  set.  Most  respondents  found  the  proposed  metric  to  be  more 
effective  in  managing  the  mission  execution  process  than  the  previous  metric.  They  also 
stated  that  the  new  metric  would  be  more  conducive  to  cross  functional  cooperation 
between  operations,  maintenance  and  logistics  readiness  units. 

Average  Delayed  Discrepancy  rate 

Comparison 

Average  Delayed  Discrepancy  Rate  is  defined  as  the  average  number  of  deferred 
maintenance  actions  (Air  Force,  2006).  Deferred  actions  are  those  maintenance  issues 
not  affecting  safety  of  flight,  which  are  selectively  kept  open  until  an  opportune  time  to 
repair  is  available.  It  is  calculated  by  capturing  a  current  snapshot  of  open  maintenance 
actions  divided  by  the  average  number  of  aircraft  possessed.  Based  on  the  key  attributes 
analysis  no  metric  was  found  in  the  SCOR  list  that  corresponded  with  this  metric.  The 
key  attributes  were  compared  to  the  key  performance  indicators  established  in  the  Oracle 
software.  Work  Order  (Maintenance  request)  Backlog  was  found  to  be  similar  to 
Average  Delay  Discrepancies  along  the  metric  definitions,  indicating  a  Level  1 
comparison  (Oracle,  2007).  Oracle  defines  Work  Order  (Maintenance  request)  Backlog 
as  the  number  of  open  maintenance  requests  on  a  selected  date.  The  calculation  is  simply 
the  snapshot  of  open  requests. 
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Figure  4.6  Average  Delayed  Discrepancy  Rate  Metric  Summary 

Importance  and  Effectiveness 

Interview  subjects  rated  Average  Delay  Discrepancies  as  moderate  in  importance  and 
effectiveness.  The  majority  of  the  interviewees  described  delay  discrepancies  as  a  strong 
indicator  of  how  well  a  unit  manages  their  airframes.  Being  able  to  effectively  manage 
and  work  these  items  will  provide  long  term  MDS  operational  readiness.  However  the 
main  issue  that  reduced  its  importance/effectiveness  ratings  was  the  fact  that  this  metric 
as  currently  defined  is  a  snapshot  in  time.  This  fails  to  provide  any  trend  analysis  of 
delay  discrepancies  to  allow  managers  to  see  patterns  or  other  long  tenn  effects.  Most 
respondents  felt  that  keeping  Delay  Discrepancies  was  important  but  the  metric  needed  to 
be  changed  to  allow  for  trending  by  airframe. 

Translation  and  Understanding 

Respondents  rated  Work  Order  (Maintenance  request)  Backlog  relatively  low  on  the 
translation  evaluation  scale.  The  responses  indicated  this  was  due  more  to  the  fact  the 
snapshot  method  was  insufficient  for  management  of  the  process,  than  the  metric  not 
translating  well.  Understanding  was  also  lower  as  respondents  had  some  difficulty 
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equating  maintenance  actions  for  maintenance  requests.  One  respondent  did  rate  the 
Translation  as  very  adequate  due  to  the  fact  the  new  measure  captures  the  metric  data  by 
individual  aircraft  instead  of  being  averaged  across  all  possessed  airframes. 

Maintenance  Scheduling  Effectiveness  Rate 

Comparison 

Maintenance  Scheduling  Effectiveness  (MSE)  is  the  percentage  of  scheduled 
maintenance  action  verses  accomplished  scheduled  maintenance  actions  based  on  the 
assigned  points  by  action  (Air  Force,  2006).  Each  scheduled  maintenance  action  is  given 
a  set  point  value  for  completion.  The  values  of  all  completed  actions  are  then  compared 
to  total  possible  values  for  the  set  time  period,  usually  a  week.  It  is  calculated  by  taking 
total  points  earned  divided  by  total  points  assigned.  Attainment  to  Plan  (Metric  190) 
from  the  SCOR  model  list  was  found  to  be  the  best  match  to  MSE  based  on  comparison 
of  the  equation,  making  it  a  Level  1  comparison  (Appendix  B).  Attainment  to  Plan  is 
defined  as  measuring  how  well  a  unit  meets  its  respective  planned  actions.  It  is 
calculated  by  dividing  the  number  of  actual  accomplished  events  by  number  of  planned 
events. 
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Figure  4.7  MSE  Rate  Metric  Summary 

Importance  and  Effectiveness 

MSE  was  rated  high  in  importance  and  effectiveness  across  respondents.  As  stated  by 
one  respondent  “there  is  nothing  any  more  important  than  scheduled  maintenance.  You 
will  cancel  your  flying  activities  to  do  your  scheduled  maintenance.”  MSE  is  a  critical 
part  of  ensuring  this  activity  is  managed  effectively.  Even  with  the  importance  placed  on 
scheduled  maintenance  by  most  respondents,  its  relative  importance  appears  to  be  MDS 
specific.  In  a  fighter  world  where  schedules  are  set  a  month  out,  MSE  will  play  a  much 
stronger  role  than  in  an  airlift  environment  where  schedules  are  set  days  or  sometimes  a 
week  out.  However,  for  maintenance  managers,  the  ability  to  maintain  a  planned 
schedule  is  critical  to  reducing  flux  within  the  process,  and  MSE  will  directly  reflect  this 


Translation  and  Understanding 

Respondents  rated  MSE  high  on  both  translation  and  understanding.  Responses  indicated 
the  new  metric  was  at  least  equivalent  to  the  current  metric.  A  few  interviews  stated  that 
they  felt  the  new  metric  was  actually  clearer  than  the  current  metric.  By  removing  the 
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element  of  the  points  system  from  the  current  metric,  this  would  make  the  metric  easier  to 
use  for  maintenance  managers. 


Cannibalization  Rate 

Comparison 

Cannibalization  Rate  or  CANN  Rate  is  the  percentage  cannibalization  actions  to  replace 
parts  on  other  aircraft.  Essentially  this  is  the  measure  that  tracks  how  often  good  parts 
are  removed  from  aircraft  to  be  used  to  repair  other  aircraft  with  a  higher  maintenance 
priority.  CANN  rate  is  calculated  by  dividing  the  total  number  of  CANN  actions  by 
Total  Sorties  flown.  Nomenclature,  Definition  and  Equation  were  all  compared  to  the 
available  metrics  in  the  SCOR  model  list  (Appendix  B).  Cannibalization  rate  (Metric 
192)  was  found  to  be  a  Level  3  comparison  by  matching  across  all  three  key  attributes. 
SCOR  defines  Cannibalization  rate  as  the  average  number  of  CANN  actions  per  100 
sorties  flown  for  flying  assets.  It  is  calculated  by  dividing  the  sum  of  the  number  of 
aircraft  to  aircraft  CANN’s  plus  number  of  engine  to  aircraft  CANN’s  by  Total  Sorties 
flown. 
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Figure  4.8  CANN  Rate  Metric  Summary 
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Importance  and  Effectiveness 

CANN  rate  was  found  to  rate  low  in  importance  and  effectiveness.  Respondents  found 
that  the  metric  was  not  effective  due  to  the  fact  that  it  had  little  effect  on  the  decision 
process.  The  study  found  that  while  cannibalizations  action  were  manpower  intensive 
based  on  the  fact  you  essentially  had  to  do  double  maintenance,  they  would  occur  no 
matter  how  many  had  been  done  before  due  to  lack  of  parts.  A  few  managers  stated  that 
CANN  rate  is  good  indicator  of  the  health  of  the  supply  support  system,  but  the  data  is 
captured  in  other  metrics  such  as  MICAP  status.  Similarly  to  MSE  this  metric  may  also 
shift  its  importance  based  on  MDS  application,  older  airframes  with  less  robust  supply 
pipelines  may  rely  on  it  more  for  maintenance  planning. 

Translation  and  Understanding 

Even  due  to  the  low  ratings  of  importance  this  metric  rated  moderately  high  for  both 
translation  and  understanding.  The  new  CANN  rate  was  viewed  as  a  better  fonnat 
providing  more  detail  into  where  cannibalizations  were  occurring.  Several  interviewees, 
while  rating  the  metrics  as  good,  felt  that  the  denominators  of  the  equations  were 
irrelevant.  A  better  comparison,  such  as  number  of  maintenance  actions  or  number  of 
aircraft  as  the  denominator  would  provide  a  more  stable  and  clearer  picture  of  the  impact 
of  cannibalization  actions. 
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Mission  Capable  Rate 

Comparison 

Mission  Capable  rate  is  the  percentage  of  hours  aircraft  are  mission  capable,  the  amount 
of  time  an  aircraft  can  do  what  it  was  designed  to  do  (Air  Force,  2007).  Mission  capable 
rate  is  calculated  by  adding  the  Fully  Mission  Capable  hours  plus  the  Partially  Mission 
Capable  hours  dividing  by  total  possessed  aircraft  hours.  After  reviewing  the  metrics 
provide  in  the  SCOR  list  (Appendix  B),  Weapon  System  Availability  (Metric  800)  was 
found  to  connect  along  two  attributes,  definition  and  equation,  making  it  a  Level  2 
comparison.  No  metric  was  found  to  match  within  the  Oracle  system.  Weapon  System 
Availability  (WSA)  is  the  percentage  of  hours  a  reported  unit  possessed  weapon  system 
was  capable  of  performing  any  assigned  mission.  It  is  calculated  by  dividing  MC  hours 
by  Total  Active  Inventory  hours. 


Current  Metric 

SCOR  Metric 

Oracle  Metric 

FMC  Hours  +  PMC  Hours 

- > 

Missou  Capable  Hours 

Total  Possessed  Hours 

N- Nomenclature 

To  tal  A  ctive  Iuveutory  H  ours 

N- Nomenclature 

Y- Definition 

N- Equation  N- Equation 


Investigative  Question  results 

Importance  Effectiveness 

Translation  Understanding 

Moderate 

Clear 

Figure  4.9  Mission  Capable  Rate  Metric  Summary 

Importance  and  Effectiveness 

Mission  Capable  rate  was  rated  as  moderate  across  the  interviews  with  some  SME’s 
rating  it  as  unimportant.  Mission  Capable  rate  was  described  as  the  “ultimate  report 
card”  for  maintenance  managers.  It  is  a  key  metric  into  providing  a  clear  picture  of  what 
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a  unit  can  provide  in  terms  of  MDS  capability.  Respondents  stated  that  MC  rate  is  core 
to  the  management  decision  process,  and  is  truly  powerful  when  combined  with  other 
metrics.  Also  mission  capable  rate  was  found  more  relevant  to  a  combat  forces  unit  than 
to  an  airlift  forces  unit.  However  the  interviews  revealed  equally  strong  opinions  on  how 
Mission  Capable  rate  was  not  important  to  maintenance  managers.  This  metric  was  often 
over  emphasized  by  senior  leadership  to  the  detriment  of  other  metrics.  Because  of  this, 
the  data  that  drives  this  metric  is  susceptible  to  manipulation  in  order  to  satisfy  some 
artificial  standard.  Managers  may  be  driven  to  make  poor  decisions  that  waste  resources 
and  manpower  on  fixing  aircraft  in  order  to  “chase”  this  metric. 

Translation  and  Understanding 

The  study  found  that  Weapon  System  Availability  rated  moderately  strong  on 
translatability.  Managers  found  the  new  metric  more  closely  aligns  with  the  strategic 
goals  from  executive  leadership.  By  being  able  to  track  what  a  unit  can  offer  in  terms  of 
weapon  systems,  a  better  picture  of  how  to  meet  taskings  is  provided.  Several 
respondents  identified  that  with  the  new  metric  having  a  different  focus  from  the  current, 
there  is  a  need  for  sufficient  training  and  education  at  all  levels  in  order  to  smoothly 
facilitate  the  transition. 
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Fully  Mission  Capable  Rate 

Comparison 

Fully  Mission  Capable  rate  is  the  percentage  of  hours  an  aircraft  can  perform  all  assigned 
missions  (Air  Force,  2007).  It  is  calculated  by  taking  total  FMC  hours  divided  by  total 
possessed  hours.  After  comparing  all  three  key  attributes  no  metric  was  found  to  match 
in  either  the  SCOR  model  or  Oracle  system.  This  metric  was  then  only  evaluated  for 
importance  and  effectives  in  the  current  system. 
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Figure  4.10  FMC  Rate  Metric  Summary 

Importance  and  Effectiveness 

Respondents  rate  Fully  Mission  Capable  as  moderate  in  importance  and  effectiveness,  but 
most  of  the  observations  discount  this  metrics  use  in  managing  aircraft.  A  few  managers 
identified  this  metric  as  important  in  its  ability  to  assist  in  assigning  airframes  to  mission 
schedules,  especially  in  a  combat  forces  unit,  where  system  requirements  are  much  more 
stringent.  However,  most  comments  stated  that  this  metric  is  rarely  used  for  management 
decisions.  It  is  a  “nice  to  look  at,  feel  good  metric”  but  the  information  provided  is 
already  captured  in  Weapon  System  Availability  better.  Additionally,  this  metric  can  be 
subjective  across  operational  environments.  An  aircraft  may  be  FMC  at  a  home  station 
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base,  but  be  PMC  or  NMC  at  a  deployed  location  for  exactly  the  same  system  capability. 
Respondents  indicated  that  they  saw  no  real  need  to  translate  FMC  into  the  new  system. 

Translation  and  Understanding 

After  reviewing  the  SCOR  metrics  list  and  Oracle  software  manuals  provided  by  the 
Logistics  Transformation  Office  (Appendix  B),  the  study  found  that  there  is  no  metric 
similar  to  fully  mission  capable  rate.  Therefore  this  metric  was  evaluated  only  on 
importance  and  effectiveness  investigative  questions. 

Partially  Mission  Capable  Rate 

Comparison 

Partially  Mission  Capable  (PMC)  Rate  is  the  percentage  of  hours  an  aircraft  can  perform 
some  but  not  all  of  its  assigned  missions  (Air  Force,  2007).  PMC  rate  is  calculated  by 
dividing  PMC  hours  by  total  possessed  hours.  As  in  the  case  of  FMC  rate  no  metric  was 
found  in  either  SCOR  or  Oracle  to  correlate.  This  metric  was  then  also  only  evaluated 
terms  of  the  current  system. 
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Figure  4.1  PMC  Rate  Metric  Summary 
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Importance  and  Effectiveness 

Partially  Mission  Capable  was  rated  moderate  in  importance  across  the  interviews.  Most 
respondents  felt  that  PMC  was  important  as  it  does  not  allow  any  hiding  of  maintenance 
conditions  on  the  airframes.  They  stated  this  was  a  critical  factor  in  whether  an  aircraft 
could  be  assigned  to  particular  missions,  thus  affecting  aircraft  availability.  If  an 
airframe  has  a  system  broken  and  cannot  do  a  portion  of  a  mission,  but  could  then 
accomplish  two  other  missions,  maintenance  schedulers  and  higher  headquarters  agencies 
need  to  know  this.  Additionally  PMC  will  give  managers  insight  into  how  well  they 
execute  maintenance  plans,  as  well  as  to  any  deficiencies  in  manning  or  support 
equipment.  Of  all  the  metrics  with  no  comparable  proposed  metric  at  present,  PMC  was 
the  one  that  managers  strongly  felt  needed  a  new  metric.  A  variation  on  Weapon  System 
Availability  that  would  account  for  partially  available  aircraft  seems  to  be  needed. 

Translation  and  Understanding 

After  reviewing  the  SCOR  metrics  list  and  Oracle  software  manuals  provided  by  the 
Logistics  Transfonnation  Office  (Appendix  B),  the  study  found  that  there  is  no  metric 
similar  to  partially  mission  capable  rate.  Therefore  this  metric  was  evaluated  only  on 
importance  and  effectiveness  investigative  questions. 

Non-Mission  Capable  Rate 

Non-Mission  Capable  (NMC)  Rate  is  the  percentage  of  aircraft  that  cannot  perfonn  any 
assigned  mission  due  to  supply  and  maintenance  or  both  (Air  Force,  2007).  It  is 


55 


calculated  by  dividing  NMC  hours  by  total  possessed  hours.  Not  Mission  Capable  Both 
(Metric  813),  from  the  SCOR  list  (Appendix  B)  was  determined  to  be  the  best  match  for 
the  current  metric.  Not  Mission  Capable  Both  is  calculated  by  dividing  the  sum  of 
NMCS  hours  and  NMCM  hours  by  total  possessed  hours.  As  a  Level  3  comparison,  it 
matched  across  all  three  key  attributes.  However  no  metric  within  the  Oracle  software 
was  found  to  match  it. 
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Figure  4.12  Non  Mission  Capable  Rate  Metric  Summary 


Importance  and  Effectiveness 

Respondents  rated  this  metric  high  across  all  evaluations.  Most  managers  found  this 
metric  was  a  good  way  to  flag  and  evaluate  the  characteristics  of  aircraft  down  time.  It 
represents  the  other  side  of  the  coin  from  MC  rate,  and  is  just  as  important.  Again,  just  as 
with  MC  rate,  they  warned  of  the  propensity  to  “chase”  numbers,  rather  than  perfonn 
effective  thoughtful  maintenance  planning. 
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Translation  and  Understanding 

NMC  measured  consistently  clear  in  translation  and  understanding  across  all  interviews. 
Respondents  found  the  proposed  change  from  the  old  to  the  new  metric  sufficient  and 
understandable.  By  being  able  to  maintain  the  same  nomenclature,  definitions  and 
clarifying  the  equation,  they  felt  it  improved  upon  the  original  metric. 

Non-Mission  Capable  Rate  Supply 

Comparison 

Non-Mission  Capable  Supply  (NMCS)  Rate  is  the  percentage  of  aircraft  that  cannot 
perfonn  any  assigned  mission  due  to  supply  (Air  Force,  2007).  Not  Mission  Capable 
Supply  (Metric  808),  from  the  SCOR  list  (Appendix  B)  was  determined  to  be  the  best 
match  for  the  current  metric.  Both  metrics  are  calculated  by  dividing  NMCS  hours  by 
total  possessed  hours.  The  only  difference  is  the  Air  Force  definition  uses  Non  while  the 
SCOR  definition  uses  Not.  As  a  Level  3  comparison,  it  matched  across  all  three  key 
attributes.  However  no  metric  within  the  Oracle  software  was  found  to  correlate. 
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Figure  4.13  NMCS  Rate  Metric  Summary 
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Importance  and  Effectiveness 

As  a  portion  of  NMCB,  Non  Mission  Capable  Supply  was  rated  as  highly  important 
across  most  interviews.  This  metric  was  found  to  be  one  of  the  critical  tools  cited  for  the 
maintenance  manager  second  only  to  Non  Mission  Capable  Maintenance.  NMCS  plays  a 
large  role  in  being  able  to  evaluate  the  robustness  of  the  supply  support  provided  by 
either  contractor  or  in  house  blue  suite  personnel.  As  a  manager  you  have  to  know  what, 
if  any,  supply  issues  may  affect  repairs  to  your  aircraft  and  be  able  to  identify  them. 

While  the  process  that  this  metric  monitors  is  controlled  by  other  entities,  its  impact  to 
the  maintenance  enterprise  makes  it  important.  Managers  cited  the  need  to  have  visibility 
on  when  needed  parts  would  arrive,  so  that  they  could  plan  around  the  arrival  dates,  thus 
better  utilizing  their  limited  manning  resources. 

Translation  and  Understanding 

NMCS  measured  consistently  clear  in  translation/understanding  across  all  interviews. 
Respondents  found  the  proposed  change  from  the  old  to  the  new  metric  sufficient  and 
understandable.  By  being  able  to  maintain  the  same  nomenclature,  definitions  and 
equation,  they  felt  it  maintained  the  essence  of  the  original  metric. 

Non-Mission  Capable  Rate  Maintenance 

Comparison 

Non-Mission  Capable  Maintenance  (NMCM)  Rate  is  the  percentage  of  aircraft  that 
cannot  perfonn  any  assigned  mission  due  to  maintenance  (Air  Force,  2007).  Not  Mission 
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Capable  Maintenance  (Metric  812),  from  the  SCOR  list  (Appendix  B)  was  determined  to 
be  the  best  match  for  the  current  metric.  Both  metrics  are  calculated  by  dividing  the  sum 
NMCM  hours  by  sum  of  possessed  hours.  As  a  Level  3  comparison,  it  matched  across  all 
three  key  attributes.  The  only  difference  is  the  Air  Force  definition  uses  Non  while  the 
SCOR  definition  uses  Not.  No  metric  within  the  Oracle  software  was  found  to  correlate. 
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Figure  4.14  NMCM  Rate  Metric  Summary 

Importance  and  Effectiveness 

Non  Mission  Capable  Maintenance  was  rated  as  highly  important  across  most  interviews. 
This  metric  was  also  cited  as  one  of  the  critical  tools  for  the  maintenance  manager. 
NMCM  plays  a  large  role  in  being  able  to  evaluate  the  robustness  of  the  maintenance 
program  at  the  unit  level,  making  it  one  of  the  cornerstone  metrics  for  maintenance 
managers.  Maintenance  mangers  stated  this  metric  was  the  most  important  as  it  was  able 
to  identify  what  was  broken,  how  often  it  was  broken  and  was  key  to  fonnulating 
maintenance  plans. 
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Translation  and  Understanding 

NMCM  measured  consistently  clear  in  translation/understanding  across  most  interviews. 
Respondents  found  the  proposed  change  from  the  old  to  the  new  metric  sufficient  and 
understandable.  By  being  able  to  maintain  the  same  nomenclature,  definitions  and 
equation,  they  felt  it  maintained  the  essence  of  the  original  metric.  Only  one  respondent 
felt  the  translation  was  not  adequate  and  did  not  think  it  was  clear  in  its  definition. 

Total  Non-Mission  Capable  Rate 

Comparison 

Total  Non-Mission  Capable  (TNMC)  Rate  is  the  percentage  of  total  NMC  hours  (Air 
Force,  2007).  It  is  calculated  by  dividing  total  NMC  Maintenance  hours  by  total 
possessed  hours.  No  metric  within  the  Oracle  software  or  the  SCOR  model  list  was 
found  to  correlate  with  TNMC. 
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Figure  4.15  TNMC  Rate  Metric  Summary 

Importance  and  Effectiveness 

TNMC  rate  was  evaluated  as  one  of  the  lowest  in  importance  and  effectiveness  by 
maintenance  managers  sampled.  This  metrics  was  found  to  be  unimportant  as  it  was  a 
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repeat  of  other  metrics,  one  respondent  stated  “I  can  do  public  math  and  can  tell  1+1=2.  I 
don’t  need  someone  to  tell  me  2  when  I  can  see  it  for  myself’.  TNMC  was  also  found  to 
be  unused  in  any  management  decision  process,  citing  the  story  was  told  better  and 
clearer  by  other  metrics. 

Translation  and  Understanding 

After  reviewing  the  SCOR  metrics  list  and  Oracle  software  manuals  provided  by  the 
Logistics  Transfonnation  Office  (Appendix  B),  the  study  found  that  there  is  no  metric 
similar  to  total  non  mission  capable  rate.  Therefore  this  metric  was  evaluated  only  on 
importance  and  effectiveness  investigative  questions. 

Total  Non-Mission  Capable  Maintenance  Rate 

Comparison 

Total  Non-Mission  Capable  Maintenance  (TNMCM)  Rate  is  the  percentage  of  total 
NMCM  hours  (Air  Force,  2007).  It  is  calculated  by  dividing  total  NMC  Maintenance 
hours  by  total  possessed  hours.  The  metric  Total  Not  Mission  Capable  Maintenance, 
from  the  SCOR  list  (Appendix  B)  was  detennined  to  be  the  best  match  for  the  current 
metric.  Total  Not  Mission  Capable  Maintenance  is  calculated  by  dividing  the  sum  of 
NMCM  plus  the  sum  of  NMCB  hours  by  sum  of  possessed  hours.  As  a  Level  3 
comparison,  it  matched  across  all  three  key  attributes.  The  one  difference  found,  is  the 
SCOR  equation  also  incorporates  any  accumulated  Both  time,  as  it  has  a  component  of 
maintenance  time  in  it.  This  metric  is  typically  used  to  aggregate  on  a  month  time  frame 
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for  a  unit  or  multiple  units  if  used  at  a  higher  headquarters.  No  metric  within  the  Oracle 
software  was  found  to  correlate. 


Current  Metric 

SCOR  Metric 

Oracle  Metric 

Total  Noil  Mission  Capable  Maintenance  Honrs 

- > 

Total  Not  Mission  Capable  Maintenance  Hours  +\ 

Total  Possessed  Homs 

Y- Nomenclature 

Total  Not  Mission  Capable  Both  Hours  J 

N-  Nomenclature 

Y- Definition 

Total  Active  Inventory  Homs 

Y- Equation  N- Equation 


Investigative  Question  results 

Importance  Effectiveness 

Translation  Understanding 

Moderate 

Gear 

Figure  4.16  TNCM  Rate  Metric  Summary 

Importance  and  Effectiveness 

TNMCM  rate,  like  TNMC,  was  identified  as  a  duplicate  metric  by  respondents,  and  was 
accordingly  rated  as  moderate  in  importance  and  effectiveness.  Having  a  second  metric 
was  a  waste  of  collection  effort  that  could  be  put  towards  other  endeavors.  They  also 
found  that  putting  all  NMC  hours  into  one  lump  did  not  allow  the  effective  analysis  by 
airframe  in  order  to  spot  aircraft  specific  trends. 

Translation  and  Understanding 

The  respondents  did  rate  this  metric  fairly  clear  in  both  translation  and  understanding. 
Respondents  found  the  proposed  change  from  the  old  to  the  new  metric  sufficient  and 
understandable.  They  found  the  new  metric  to  actually  be  better,  in  that  it  incorporated 
the  NMCB  time  which  contains  some  maintenance  time  which  the  current  metric  did  not 
capture. 
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Total  Non-Mission  Capable  Supply  Rate 

Total  Non-Mission  Capable  Supply  (TNMCS)  Rate  is  the  percentage  of  total  NMCS 
hours  (Air  Force,  2007).  It  is  calculated  by  dividing  total  NMC  Supply  hours  by  total 
possessed  hours.  Total  Not  Mission  Capable  Supply  (Metric  811),  from  the  SCOR  list 
(Appendix  B)  was  determined  to  be  the  best  match  for  the  current  metric.  Total  Not 
Mission  Capable  Supply  is  calculated  by  dividing  the  sum  of  NMCS  plus  the  sum  of 
NMCB  hours  by  sum  of  possessed  hours.  The  one  difference  found,  is  the  SCOR 
equation  also  incorporates  any  accumulated  Both  time,  as  it  has  a  component  of  supply 
time  in  it.  This  metric  is  typically  used  to  aggregate  on  a  month  time  frame  for  a  unit  or 
multiple  units  if  used  at  a  higher  headquarters.  As  a  Level  3  comparison,  it  matched 
across  all  three  key  attributes.  However  no  metric  within  the  Oracle  software  was  found 
to  match. 
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Figure  4.17  TNMCS  Rate  Metric  Summary 

Importance  and  Effectiveness 

TNMCS  rate,  like  TNMCM,  was  identified  as  a  duplicate  metric  by  respondents,  and  was 
accordingly  rated  as  low  in  importance  and  effectiveness.  Having  a  second  metric  was  a 
waste  of  collection  effort  that  could  be  put  towards  other  endeavors.  The  metric  doesn’t 
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provide  sufficient  detail  to  support  management  decisions.  One  interviewee  felt  the 
metric  “would  miss  the  trees,  in  spite  of  the  forest”.  While  this  metric  could  be  used  to 
influence  CANN  decisions,  the  same  infonnation  was  also  garnered  from  the  NMCS  rate. 

Translation  and  Understanding 

The  respondents  did  rate  this  metric  clear  in  both  translation  and  understanding. 
Respondents  found  the  proposed  change  from  the  old  to  the  new  metric  sufficient  and 
understandable.  They  found  the  new  metric  to  actually  be  better  with  its  incorporation  of 
the  NMCB  time,  which  contains  some  supply  time  that  the  current  metric  did  not  capture. 

Flying  Hour  Execution  Rate 

Comparison 

Flying  Hour  Execution  (FHE)  is  the  percentage  of  actual  flying  hours  executed  verses  the 
planned  flying  hours  (Air  Force,  2007),  and  is  calculated  by  taking  the  total  flying  hours 
executed  divided  by  total  flying  hours  scheduled.  Operational  Programmed  Time 
Accuracy,  (OPTA)  from  the  SCOR  list  (Appendix  B)  was  found  to  be  a  Level  2 
comparison,  matching  on  definition  and  equation.  OPTA  is  the  comparison  of  future 
Operational  Time  (i.e.  scheduled  flying  hours)  to  actual  Operational  Time  (i.e.  executed 
flying  hours)  by  location  or  group.  It  is  calculated  by  dividing  actual  executed  flying 
hours  by  programmed  flying  hours.  No  metric  was  found  in  the  Oracle  listing  to  match 
to  FHE. 
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Figure  4.18  FHE  Rate  Metric  Summary 

Importance  and  Effectiveness 

FHE  is  one  of  the  metrics  where  SME’s  were  split  on  rating  the  importance  and 
effectiveness,  but  this  metric  was  rated  low  overall.  The  split  occurred  between 
application  in  a  combat  aircraft  unit  and  a  mobility  aircraft  unit.  Fighter  units  placed  a 
high  value  on  this  metric,  while  airlift  units  almost  never  use  this  metric.  In  a  high 
importance/effectiveness  environment  this  metric  dictates  all  aspects  of  management 
planning.  Flying  schedules  and  maintenance  planning  are  done  month  by  month  and  any 
over  or  under  flying  will  add  flux  into  these  schedules.  Flux,  in  turn,  means  more  effort 
by  maintenance  personnel  to  compensate  for  the  changes.  FHE  also  determines  budget 
allocation  in  the  fighter  community.  So  executing  on  target  is  critical  for  effective 
utilization  and  acquisition  of  operations  and  maintenance  funds.  In  direct  contrast,  within 
the  airlift  community  the  majority  of  flying  hours  are  dictated  by  the  Tanker  and  Airlift 
Control  Center  based  on  mission  execution  needs,  so  managers  in  airlift  have  found  little 
use  for  this  metric  at  the  unit  level. 
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Translation  and  Understanding 

This  study  found  that  the  respondents  were  divided  on  rating  the  translation  and 
understanding.  While  some  members  clearly  understood  the  new  proposed  metric  and 
felt  the  translation  was  adequate,  others  did  not.  The  language  used  in  both  the  definition 
and  equation  was  identified  as  the  largest  issue  with  understanding  the  new  metric. 
Respondents  cited  confusion  in  trying  to  understand  the  terminology  of  the  SCOR 
definition  and  how  it  related  to  their  understanding  of  the  Air  Force  definition.  They 
identified  a  need  for  strong  training  and  education  in  order  to  comprehend  this  metric. 

Flying  Scheduling  Effectiveness  rate  and  Chargeable  Deviation  rate 

Comparison 

Flying  Scheduling  Effectiveness  (FSE)  is  the  percentage  of  deviations  from  the  planned 
flying  schedule  to  what  was  actually  flown,  and  is  calculated  by  subtracting  schedule 
deviations  from  scheduled  flights  then  divided  by  actual  executed  flights.  Chargeable 
Deviation  rate  is  the  percentage  of  deviations  due  to  maintenance  or  operations  actions, 
calculated  by  dividing  the  sum  of  maintenance  deviations  and  operations  deviations  by 
total  sorties  scheduled  (Air  Force,  2007).  Chargeable  Deviation  is  essentially  a  derivative 
of  FSE  that  reports  on  two  possible  deviation  types;  because  of  this  the  study  chose  to 
combine  these  metrics.  Based  on  their  key  attributes,  Planning  Changes  within  Execution 
Window  (Metric  181)  was  determined  to  be  a  Level  2  comparison.  Planning  Changes 
within  Execution  Window  is  the  number  of  changes  to  a  schedule  within  the  execution 
window.  While  labeled  as  a  rate  this  metric  currently  is  a  count  of  event  deviations.  This 
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metric  can  be  converted  to  a  rate  by  dividing  number  of  events  minus  number  of  changes 
by  number  of  total  events. 


Current  Metrics 
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- » 

Actual  Flying  Schedule  Events 
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Figure  4.19  FSE  and  CD  Rate  Metric  Summary 

Importance  and  Effectiveness 

Overall  FSE  and  Chargeable  Deviation  rates  were  marked  as  low  for  both  importance  and 
effectiveness.  Some  of  the  respondents  had  never  even  heard  of  chargeable  deviation 
outside  of  the  Air  Force  instruction  manuals.  This  metric  was  felt  to  be  more  important 
to  senior  managers  than  to  folks  at  lower  tactical  levels.  When  used  FSE  can  provide  a 
fairly  accurate  report  card  on  how  units  execute  their  given  taskings,  the  key  being  when 
used.  The  metric  received  more  attention  in  a  fighter  operating  environment  than  a 
mobility  one.  It  was  also  observed  that  FSE  also  may  not  tell  the  full  story  of  launch  or 
recovery  operations. 


Translation  and  Understanding 

Case  study  interviews  also  showed  that  FSE/CD  was  rated  moderate  compared  to  other 
metrics  for  translation  and  understanding.  However,  between  respondents  this  metric  set 
was  split  in  its  ratings,  it  was  rated  either  to  the  right  of  the  scale  or  the  left,  with  no 
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ratings  in  the  middle.  Low  rated  observations  felt  that  some  fidelity  was  lost  in  the 
combination  of  the  two  metrics,  and  data  on  deviations  would  not  be  able  to  be  extracted. 
Managers  who  rated  the  translation  high  felt  the  new  metric  captured  more  of  the  process 
events  instead  of  just  whether  the  aircraft  made  it  off  on  time.  If  the  aircraft  took  off  on 
time,  but  each  step  in  the  Schedule  of  Events  busted  time,  you  would  then  see  the  flux  in 
the  system  and  be  able  to  better  identify  where  resources  need  to  be  applied. 

Primary  Aircraft  Inventory 

Comparison 

Primary  Aircraft  Inventory  (PAI)  is  the  assigned  number  of  aircraft  by  MDS  per 
designated  organization.  This  is  simply  a  snapshot  of  the  number  of  aircraft  assigned. 
PAI  is  used  in  comparison  to  Possessed  Aircraft  Rate  to  detennine  a  unit’s  ability  to  meet 
assigned  taskings.  The  metric  Total  Active  Inventory  (TAI)  was  found  to  match  along 
two  key  attributes,  setting  this  at  a  Level  2  comparison.  Total  Active  Inventory  is  the 
possessed  number  of  aircraft  by  weapon  system  with  possession  purpose  code  =  CA,  CB, 
CC,  CF,  EH,  El,  IF,  PJ,  PL,  PR,  TF,  TJ,  ZA,  ZB.  Possession  purpose  codes  are  the  two 
letter  codes  within  the  maintenance  data  systems  that  identify  who  the  controlling 
organization  is  for  aircraft.  The  codes  listed  above  all  indicated  possession  of  an  aircraft 
by  the  base  level  organization.  As  with  PAI  it  is  simply  a  snapshot  count  of  assets 
possessed. 
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Figure  4.20  PAI  Rate  Metric  Summary 

Importance  and  Effectiveness 

PAI  was  rated  the  lowest  metric  in  importance  and  effectiveness  by  interview 
respondents.  The  primary  reason  stated  for  the  low  rating  was  the  fact  that  at  the  unit 
level  there  is  no  control  over  this  metric  and  it  is  not  used  for  any  decision  level.  PAI  is 
set  by  higher  headquarters  and  directed  down  to  units.  PAI  was  also  marked  as  not 
effective  because  it  only  tells  part  of  the  story  for  the  data  it  reports  on.  PAI  is  the 
number  of  aircraft  that  higher  headquarters  use  to  establish  manning  and  equipment 
support  levels.  However  many  units  also  have  more  airframes  assigned  than  PAI 
indicates,  this  excess  is  termed  Backup  Aircraft  Inventory  (BAI).  BAI  aircraft  while 
being  utilized  just  like  PAI  aircraft  do  not  get  any  manning  or  equipment  authorizations, 
and  BAI  counts  are  not  utilized  in  metric  decisions.  So  in  essence  units  may  operate 
more  systems  than  they  are  equipped  or  manned  to  support.  One  respondent  stated  he 
had  never  seen  a  unit  have  more  than  its  PAI  number  of  aircraft.  Additionally  TAI  also 
comprises  part  of  the  many  other  metrics  equations  discussed  earlier. 
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Translation  and  Understanding 

Respondents  did  rate  TAI  clear  for  both  Translation  and  Understanding.  The  possession 
codes  did  cause  some  respondents  to  not  understand  the  new  metric  at  first,  as  they  did 
not  know  or  understand  the  purpose  of  the  possession  codes.  However,  most 
respondents  thought  the  new  metric  was  better  because  it  eliminated  the  PAI/BAI 
conflict.  The  new  metric  tracked  what  you  own  and  have  to  operate,  which  they  felt  was 
a  better  way  of  tracking  true  utilization. 

Possessed  Aircraft  Rate 

Comparison 

Possessed  Aircraft  Rate  is  the  aircraft  under  control  of  an  owning  organization.  It  is 
simply  the  snapshot  of  aircraft  at  an  organization  for  a  given  time  window.  Possessed 
Aircraft  Rate  is  used  in  comparison  to  PAI  to  detennine  a  unit’s  ability  to  meet  assigned 
taskings.  Depot  Possessed  Percentage  rate  (Metric  815),  although  an  inverse  of 
Possessed  Aircraft  rate,  was  found  to  match  across  all  three  key  attributes,  warranting  a 
Level  3  comparison.  Depot  Possessed  Percentage  rate  is  the  percentage  of  the  total 
active  inventory  hours  that  are  in  depot  possessed  status,  calculated  by  month  for 
possession  purpose  codes  =  DJ,  DK,  DL,  DM,  DO,  DR. 
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Figure  4.21  Possessed  Aircraft  Rate  Metric  Summary 

Importance  and  Effectiveness 

Possessed  Aircraft  rate  was  also  rated  low  by  the  interviewed  SME’s.  While  a  few 
managers  felt  this  metric  was  important  from  a  scheduling  perspective  for  forecasting 
future  mission  scheduling,  most  managers  never  used  this  metric  for  any  decision  making 
processes.  This  is  a  metric  that  is  controlled  at  a  strategic  level  and  plays  no  real  role  at  a 
tactical  execution  unit.  Additionally,  respondents  identified  that  policy  places  a  cap  on 
how  many  aircraft  can  be  in  depot  status.  This  may  also  lead  to  masking  of  true 
capabilities  of  units  if  there  is  a  need  for  more  airframes  to  be  in  depot  repair  possession 
than  policy  would  allow. 


Translation  and  Understanding 

While  the  current  metric  was  rated  low  in  importance  and  effectiveness,  it  rated  high  on 
translation  and  understanding.  Maintenance  managers  felt  that  even  with  the  inverse  flip 
in  reported  data,  the  metric  was  easily  understood  and  the  proposed  metric  was  adequate. 
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Programmed  Average  Sortie  Duration 

Comparison 

Programmed  Average  Sortie  Duration  (PASD)  is  the  measure  of  the  average  sortie  length 
scheduled  for  execution.  It  is  calculated  by  dividing  the  number  of  sorties  planned  by 
total  scheduled  sorties  hours  (Air  Force,  2007).  After  reviewing  the  available  metrics  in 
both  the  SCOR  listing  in  Appendix  B  and  the  Oracle  software  manuals,  no  metric  was 
found  to  correlate  with  PASD.  This  metric  was  only  evaluated  by  managers  on 
importance  and  effectiveness  of  the  current  metric. 
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Figure  4.22  PASD  Rate  Metric  Summary 

Importance  and  Effectiveness 

PASD  was  another  metric  that  was  found  to  have  a  split  evaluation  among  the 
maintenance  managers  interviewed.  Several  respondents  found  this  metric  to  be  highly 
important  and  effective  if  you  operate  in  combat  forces  organization,  but  not  important  or 
effective  in  mobility  forces  organization.  Combat  forces  sorties  can  last  1  to  2  hours  with 
multiple  launches  in  a  day.  In  contrast,  in  the  mobility  airlift  organizations  sorties  can 
last  anywhere  from  4  hours  to  more  than  14  hours  when  the  aircraft  leave  home  station. 
PAS  was  identified  as  a  strong  tool  for  monitoring  and  managing  mission  execution, 
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specifically  the  flying  schedule.  In  addition  it  was  also  found  to  play  a  large  role  in 
personnel  management.  Being  able  to  know  when  to  provide  launch  and  recovery 
support  was  identified  as  one  of  the  core  responsibilities  of  maintenance  management. 
Respondents  stated  that  this  metric  should  have  a  translation  built  for  it  in  the  ECSS 
system  program  transition. 

Translation  and  Understanding 

After  reviewing  the  SCOR  metrics  list  and  Oracle  software  manuals  provided  by  the 
Logistics  Transformation  Office  (Appendix  B),  the  study  found  that  there  is  no  metric 
similar  to  PASD.  Therefore  this  metric  was  evaluated  only  on  importance  and 
effectiveness  investigative  questions. 

Actual  Average  Sortie  Duration 

Comparison 

Actual  Average  Sortie  Duration  (AASD)  is  the  measure  of  the  average  sortie  length 
executed  by  a  unit.  It  is  calculated  by  dividing  the  number  of  sorties  executed  by  total 
executed  sorties  hours  (Air  Force,  2007).  After  reviewing  the  available  metrics  in  both 
the  SCOR  listing  in  Appendix  B  and  the  Oracle  software  manuals,  no  metric  was  found 
to  correlate  with  AASD.  This  metric  was  only  evaluated  by  interview  on  importance  and 
effectiveness  of  the  current  metric. 


73 


Current  Metric 

SCOR  Metric 

Oracle  Metric 

ft  of  Sorties  Executed 

X 

Total  Executed  Sortie  Hours 

N-  Nomenclature 

N-  Nomenclature 

N- Equation  N-  Equation 


Investigative  Question  results 

Importance  |  Effectiveness 

Translation  |  Understanding 

Moderate 

N/A 

Figure  4.23  AASD  Rate  Metric  Summary 

Importance  and  Effectiveness 

AASD  was  also  found  to  have  a  split  evaluation  among  the  maintenance  managers 
interviewed.  The  same  respondents  that  found  PASD  important  also  rated  this  metric  as 
highly  important  and  effective  if  you  operate  in  a  combat  forces  organization,  but  not 
important  or  effective  in  mobility  forces  organization.  For  the  same  reasons  stated  for 
PASD,  maintenance  managers  identified  this  metric  as  being  ineffective  due  to 
incompatibility  of  data  exchange  between  maintenance  systems  and  operations  systems. 
Respondents  stated  that  this  metric  should  have  a  translation  built  for  it  in  the  ECSS 
system  program  transition. 

Translation  and  Understanding 

After  reviewing  the  SCOR  metrics  list  and  Oracle  software  manuals  provided  by  the 
Logistics  Transformation  Office  (Appendix  B),  the  study  found  that  there  is  no  metric 
similar  to  AASD.  Therefore  this  metric  was  evaluated  only  on  importance  and 
effectiveness  investigative  questions. 
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Utilization  Rate 


Comparison 

Utilization  (UTE)  Rate  is  the  average  number  of  sorties  flown  per  primary  aircraft 
inventory,  calculated  by  dividing  number  of  sorties  by  hours  flown  per  month  then 
dividing  by  PAI  per  month  (Air  Force,  2007).  Utilization  rate  (Metric  766)  from  the 
SCOR  metric  model  was  found  to  match  across  all  three  key  attributes,  a  Level  3 
comparison.  Utilization  rate  is  defined  as  the  measure  of  the  average  use  of  a  system 
during  a  specified  period  of  calendar  time.  It  is  calculated  as  the  average  usage  divided 
by  flying  hours  divided  by  TAI  for  the  month. 
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Figure  4.24  UTE  Rate  Metric  Summary 

Importance  and  Effectiveness 

UTE  rate  was  rated  moderately  high  by  the  majority  of  interviews.  UTE  rate  was  a  key 
tool  cited  to  provide  critical  information  on  fleet  mission  scheduling  management, 
especially  in  the  fighter  community.  In  combat  forces  UTE  is  a  driving  factor  for  when 
aircraft  enters  scheduled  maintenance  and  overhaul,  which  is  dictated  by  flying  hours 
accumulated  on  the  airframes.  In  the  mobility  community  this  is  not  applicable  as 
scheduled  maintenance  programming  is  based  on  a  calendar  time  not  flying  hour  targets. 
However,  it  was  identified  as  being  important  at  the  strategic  level  for  fleet  management 
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and  ensuring  longevity  of  airframes  across  units.  It  will  also  determine  where  allocation 
of  assets  needs  to  occur.  High  UTE  rates  mean  more  flying,  requiring  more  airframes 
and  more  manning  to  be  able  to  meet  mission  taskings. 

Translation  and  Understanding 

UTE  rate  measured  consistently  clear  in  translation/understanding  across  most 
interviews.  Respondents  found  the  proposed  metric  sufficient  and  understandable.  By 
being  able  to  maintain  the  same  nomenclature,  definitions  and  clarifying  the  equation, 
they  felt  it  improved  upon  the  original  metric. 

MICAP  Hours 

Comparison 

MICAP  or  Mission  Capability  Aircraft  Parts  Hours  are  the  hours  that  are  accrued  in  a 
given  month  for  items  designated  as  affecting  mission  capability  that  are  on  backorder. 
This  is  metric  817  from  the  SCOR  database  list.  This  metric  is  not  a  translation  of  any 
current  metric  but  was  evaluated  on  its  potential  use  if  introduced  to  the  system. 
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Figure  4.25  MICAP  Rate  Metric  Summary 
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Importance  and  Effectiveness 

Respondents  found  this  metric  could  be  important  and  effective  if  incorporated  into  the 
ECSS  program.  It  was  rated  high  in  most  interviews.  MICAP  Hours  was  identified  as 
providing  greater  visibility  into  the  MICAP  issue  process,  than  current  metrics.  If 
managed  properly  it  would  assist  maintenance  managers  in  making  CANN  process 
decisions  as  well  as  encourage  interfacing  with  supply  process  managers.  However,  one 
respondent  felt  this  data  was  already  tracked  and  captured  in  the  NMCS  rate. 

Translation  and  Understanding 

This  metric  was  a  new  metric  identified  for  evaluation.  It  was  not  translated  from  an 
existing  metric.  Therefore  this  metric  was  evaluated  only  on  importance  and 
effectiveness  investigative  questions. 

Average  Age  of  ETAR 

Average  Age  or  Engineering  Technical  Assistance  Request  is  the  average  by  unit  of  time 
of  open  requests  for  engineering  assistance,  also  known  as  107  or  Request  for 
Engineering  Disposition  (REDI)  requests.  This  is  metric  899  from  the  SCOR  database 
provided  by  the  LTO  office.  This  metric  is  not  a  translation  of  any  current  metric  but  was 
evaluated  on  its  potential  use  if  introduced  into  the  system. 
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Figure  4.26  Average  Age  of  ETAR  Rate  Metric  Summary 

Importance  and  Effectiveness 

Average  Age  of  ETAR  was  evaluated  high  among  most  interviewed  managers.  This 
metric  would  be  able  to  provide  the  case  for  system  mangers  to  increase  budgets  for 
engineering  support  as  well  as  provide  justification  to  contract  supported  units  for 
continuation  of  those  contracts.  This  would  also  create  visibility  at  a  strategic  level  of  the 
reliance  the  current  force  structure  has  on  civilian  and  contract  engineers. 

Translation  and  Understanding 

This  metric  was  a  new  metric  identified  for  evaluation.  It  was  not  translated  from  an 
existing  metric.  Therefore  this  metric  was  evaluated  only  on  importance  and 
effectiveness  investigative  questions. 

Analysis  of  Evaluation  Factors 

After  discussion  of  each  individual  metric  case,  the  study  sought  to  define  the  underlying 
themes  or  factors  that  influenced  respondent’s  evaluations  of  the  studied  metrics. 

Through  systematic  coding  analysis  of  the  interview  results,  the  researcher  indentified 
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several  themes  that  carried  through  multiple  metrics.  Evaluation  factors  were  found  to 
fall  into  two  major  categories:  Process  Management  and  Valuation  Characteristics. 
Process  Management  factors  are  the  underlying  themes  identified  by  respondents  that 


metrics  impacts  in  their  use.  Valuation  Characteristic  factors  are  underlying  themes 
identified  by  respondents  that  impact  the  value  of  a  metric.  Figure  4.30  shows  the 
relationship  of  these  to  factors  to  performance  measures. 


Figure  4.30  Evaluation  Factor  Relation  Diagram 

Each  factor  was  evaluated  on  content  and  how  they  impacted  or  were  impacted  by 
metrics.  Additionally,  each  factor  was  evaluated  on  its  role  in  a  metric  achieving  the 
three  goals  of  metrics:  drive  behavior,  create  understanding,  and  lead  to  improvement. 
Each  of  these  factor  categories  and  sub-categories  will  be  discuss  in  the  following 
sections.  Finally,  a  summary  analysis  of  the  overall  distribution  of  the  factors  in  relation 
to  their  influence  on  achieving  the  three  metric  goals  is  discussed.  The  coding  factor 
outline  can  be  found  in  Appendix  C. 

Process  Management  Factors 

Process  Management  factors  are  the  areas  within  the  maintenance  enterprise  that  metrics 
play  a  key  role  in  managing  and  overseeing.  These  are  the  factors  that  respondents 
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identified  as  areas  that  metrics  directly  impact,  either  positively  or  negatively.  The 
distribution  of  each  Process  Management  factor  category  within  the  selected  metrics  is 
shown  in  Table  4.4.  The  categories  were  sorted  in  order  of  most  mentioned  at  50  times  to 
least,  at  1  time.  Each  category  is  defined  and  evaluated  for  relevance  to  the  3  goals  of 
metrics  below. 

Table  4.1,  Process  Management  Factor  Distribution 
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Mission  Execution  Management 

Mission  Execution  Management  factors  were  cited  50  times  by  respondents.  Mission 
Execution  is  the  process  category  that  was  associated  with  metrics  that  have  an  impact  on 
the  scheduling,  launch,  and  recovery  of  aircraft  to  meet  flying  taskings.  The  ultimate  job 
of  any  Air  Force  unit  is  to  launch  aircraft.  Maintainers  must  be  able  to  measure  and 
analyze  the  aircraft  generation  process. 
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The  effects  of  metrics  on  this  category  are  diverse.  Metrics  identified  with  this  process 
provide  oversight  into  meeting  flying  plans,  as  well  as  to  be  able  to  forecast  any  major 
hiccups  that  occur  do  to  over  or  under  flying  the  plan.  For  example,  the  use  of  UTE  rate 
and  Flying  Hour  Execution  in  the  fighter  unit  is  used  determine  whether  to  input  surge  or 
down  days  to  adjust  flying  hours.  It  was  also  cited  in  metrics  that  allowed  managers  to 
detennine  how  to  allocate  aircraft  to  mission  profiles.  Break  rate,  PMC  rate  and  others 
were  found  to  be  able  to  identify  limitations  on  aircraft  that  would  not  allow  them  to  be 
assigned  certain  missions.  The  ability  to  look  back  at  an  executed  plan  to  determine  areas 
of  improvement  was  mentioned  by  several  respondents. 

Metrics  that  were  identified  with  the  Mission  Execution  Management  process  appear  to 
meet  the  three  goals  of  performance  measures.  The  metrics  allows  managers  to 
understand  the  overall  mission  execution  process,  where  they  can  make  the  appropriate 
decisions  in  order  to  meet  strategic  goals,  and  in  turn  make  improvements  within  this 
process. 

Maintenance  Planning  Management 

Maintenance  Planning  Management  was  mentioned  3 1  times  as  a  process  impacted  by 
metrics.  Metrics  that  influenced  this  factor  were  identified  as  providing  insight  and 
control  over  how  managers  planned  for  scheduled  and  unscheduled  maintenance. 
Indicators  of  how  well  managers  executed  response  management  of  maintenance 
activities,  as  well  provide  key  points  to  decision  making. 
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Respondents  cited  Fix  rate  and  Repeat/Recur  rate  as  metrics  that  affected  this  theme.  By 
being  able  to  track  these  rates  data,  managers  could  effectively  detennine  what  resources 
they  would  need  to  work  new  maintenance  issues.  Being  able  to  prioritize  maintenance 
actions  was  also  identified  as  a  benefit  of  metrics  like  NMCS  and  CANN  rates.  These 
metric  were  stated  to  provide  key  information  on  the  best  way  to  accomplish  tasks  in 
order  to  minimize  unnecessary  maintenance.  Being  able  to  plan  maintenance  effectively 
and  stay  to  the  plans  was  found  to  have  underlying  impacts  to  most  other  management 
process,  such  as  mission  execution  and  manning  utilization  management. 

As  with  Mission  Execution,  metrics  that  align  with  the  Maintenance  Planning  process 
appear  to  meet  the  three  goals  of  performance  measures.  The  metrics  give  managers 
control  and  insight  into  this  process,  this  aligns  the  decision  behaviors  with  established 
strategic  goals,  and  has  allowed  managers  to  practice  continuous  improvement  within  the 
process. 

Mission  Design  Series  Management 

Mission  Design  Series  (MDS)  Management  was  mentioned  27  times  as  a  process 
impacted  by  the  evaluated  metrics.  MDS  management  is  the  process  of  executing 
maintenance  plans  by  air  frame  type.  Respondents  categorized  the  impact  of  metrics  by 
mission  type,  either  Combat  Air  Forces  (CAF)  which  include  F-15,  F-16,  B-l  airframes 
or  Mobility  Air  Forces  (MAF)  which  operate  C-17,  C-5,  or  KC-135  aircraft. 

Additionally,  analysis  revealed  this  process  was  also  categorized  by  system  age,  divided 
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between  newer  and  older  aircraft.  Table  4.2  provides  a  breakdown  of  importance  as 
attributed  to  fighter  units  or  airlift  units.  The  metrics  identified  were  ones  that  were 
specifically  given  split  ratings  by  respondents  as  applied  to  Combat  Air  Forces  and 
Mobility  Air  Forces. 


Table  4.2  Metric  MDS  Importance 


Combat  Air  Forces 

Not  Important - >  Important 

Flying  Scheduling  Effectivness 
Programmed  Average  Sortie  Dilation 
Actual  Average  Sortie  Diration 
Utilization  Rate 

Fully  Mission  Capable 

Flying  Hoir  Execution 
Maintenance  Scheduling  Effectiveness 

Partially  Mission  Capable 
Logistics  Depart  ire  Reliability 

Mobility  Air  Forces 

Metrics  that  were  identified  with  the  MDS  management  process  were  usually  found  with 
a  split  rating.  These  splits  would  occur  either  between  MAF  application  and  CAF 
application  or  Older  Airframes  and  Newer  Airframes.  Metrics  such  as  Fix  rate,  CANN 
rate,  and  Abort  rate  were  given  much  high  ratings  for  older  systems  such  as  the  C-141  or 
the  C-5  versus  the  C-17.  For  example,  one  respondent  described  the  use  of  Abort  rate: 
“In  C-5s  you  live  your  life  around  [the  question]  is  the  aircraft  an  hour  out  from  home 

station?  If  not  it  could  still  be  coming  back.  Whereas,  with  the  C-17,  if  it  breaks  ground 
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it  is  gone”.  Additionally  the  nature  of  systems  on  aircraft  also  was  cited  to  impact  this 
process.  Larger  airframes  were  cited  as  having  more  redundant  systems  than  small 
aircraft.  This  made  them  less  susceptible  to  events  tracked  by  Abort  rate  or  Repeat/Recur 
rate.  How  taskings  are  accomplished  was  cited  as  factors  for  MDS  management 
processes.  Similar  to  Mission  Execution,  various  mission  profiles  are  programmed 
differently.  The  CAF  environment  has  a  more  rigid  fix  process,  where  schedules  are 
created  weeks,  even  months  in  advance,  making  metrics  like  FSE  and  MSE  more 
important.  In  the  MAF  world  schedules  are  more  fluid  and  don’t  coalesce  until  days  and 
even  hours  before  execution,  therefore  little  attention  is  given  to  MSE  or  FSE. 

The  metrics  identified  with  MDS  Management  are  split  on  achieving  the  3  goals  of 
metrics.  While  across  each  MDS  sub-category  the  metrics  provide  understanding,  they 
are  not  utilized  for  driving  behavior  the  same.  In  CAF  forces,  metrics  drive  the  required 
behaviors  in  response  to  the  processes,  but  in  MAF  they  have  less  impact.  While 
regardless  of  MDS,  maintenance  managers  look  for  improvements,  the  metrics  impacting 
this  process  are  favored  more  in  CAF  than  MAF  environments.; 

Manning  Utilization  Management 

Manning  Utilization  Management  was  identified  26  times  within  the  interviews. 

Manning  Utilization  management  is  the  process  by  which  maintenance  supervisors 
effectively  put  their  people  to  work.  Metrics  that  impact  Manning  Utilization  processes 
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provides  managers  the  information  to  effectively  assign  personnel  in  order  to  maximize 
their  capabilities  to  fix  aircraft. 

Respondents  cited  this  process  as  the  biggest  player  in  planning  maintenance  personnel 
task  assignments.  Metrics  like  Repeat/Recur  rate,  MICAP  rate,  MSE  rate  provide 
identifiers  on  where  supervisors  need  to  place  people,  whether  by  aircraft  job  or  by  work 
shift.  For  example,  if  aircraft  mission  windows  shift  from  day  operation  to  night,  then 
manning  needs  to  be  adjusted  to  account  for  heavier  recovery  activity  during  the  early 
morning  as  opposed  to  the  midnight  shift.  It  also  allows  managers  to  evaluate  skill 
training  and  ensure  maintainers  have  all  necessary  equipment  to  complete  any  assigned 
tasks.  Metrics  such  as  Fix  rate  and  ETAR  rate  that  impact  the  Manning  Utilization 
process  will  provide  insight  into  where  to  best  apply  limited  training  and  equipment 
resources  to  get  the  most  improvement. 

Overall  assessment  finds  that  metrics  impacting  this  category  align  with  the  three  goals  of 
metrics.  These  metrics  give  managers  an  understanding  of  how  to  best  manage  personnel 
under  them.  They  can  provide  information  on  area  than  may  need  improvement,  and  will 
drive  the  decision  process  to  achieve  these  improvements. 

Support  Management  Process 

Support  Management  was  found  to  be  a  factor  that  also  had  split  ratings.  The  study 
found  17  instances  discussed  within  the  data.  Support  Management  is  the  process  of 
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managing  the  organizations  and  programs  that  directly  support  the  maintenance 
enterprise,  but  are  not  necessarily  controlled  by  the  maintenance  manager. 

Under  Support  Management,  the  respondents  identified  that  while  they  do  not  control 
these  processes,  the  metrics  associated  with  them  allowed  them  to  plan  for  and  influence 
the  process  from  a  customer  perspective.  Respondents  identified  metrics  like  MICAP 
hours,  NMCS  rate  and  CANN  rate  as  key  measurements  to  monitor  and  hold  accountable 
upstream  supply  organizations.  For  Example,  managers  must  know  how  effective  the 
parts  supply  process  is  in  order  to  make  good  decisions  on  CANN  priorities,  in  order  to 
avoid  unnecessary  maintenance.  This  process  theme  was  often  cited  in  conjunction  with 
the  Valuation  Characteristic  Process  Control  Value,  specifically  the  NMCS  rate.  While 
respondents  had  to  track  and  report  on  this  metric  they  had  little  to  no  control  over  the 
process. 

Additionally,  metrics  that  impact  Support  Management,  like  ETAR  rate,  were  also  cited 
as  being  able  to  provide  oversight  into  how  third  party  contractors  are  executing  their 
designated  contract  responsibility.  This  is  important  for  senior  manager’s  decisions  on 
contract  negotiations  and  cancellations,  as  well  detennination  of  any  fiscal  bonuses  or 
penalties.  But  like  the  NMCS  rate,  this  decision  process  was  cited  as  beyond  the  purview 
of  the  respondents,  so  were  rated  as  less  important. 

Overall,  metrics  connected  with  the  Support  Process  Management  were  found  to  meet  the 
criteria  for  good  metrics  at  a  more  strategic  level.  They  allowed  senior  level  decision 
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makers  and  weapon  system  managers  to  closely  monitor  the  affected  processes,  thus 
allowing  them  to  make  the  appropriate  decision  behaviors  that  lead  to  improvement 
within  the  enterprise.  However  from  the  respondent’s  point  of  view,  these  metrics  only 
provide  understanding  of  processes,  not  control  or  ability  to  make  improvements. 

Budget  Management  Process 

Budget  Management  was  only  identified  2  times  in  the  interviews  as  a  process  factor. 
Budget  Management  is  the  process  of  managing  a  units  flying  hour  program  as  it  related 
to  distributed  operation  and  maintenance  (O&M)  funds.  Essentially,  for  aircraft  units  that 
operate  almost  entirely  on  O&M  money,  flying  hours  equals  to  dollars.  Emphasis  on 
this  process  factor  seems  to  be  mostly  identified  with  Combat  Air  Force  units.  Mobility 
Air  Force  units,  while  having  O&M  funds,  primarily  acquire  and  spend  budget  from 
other  sources,  such  as  real  world  mission  taskings.  Customers  using  MAF  assets  pay  for 
them,  which  then  is  added  to  the  unit’s  accounts  to  recoup  mission  costs. 

One  respondent  stated  best  the  impact  metrics  like  FHE  have  on  the  Budget  Management 
Process:  “For  the  CAF  guy  tied  to  the  O&M  budget  -  flying  hours  =  money”.  FHE  was 
the  only  metric  where  this  theme  was  identified.  It  is  critical  for  aircraft  managers  in  a 
CAF  unit  to  effectively  control  the  flying  hour  execution  process  to  ensure  they  remain 
on  target  in  order  to  meet  all  established  goals  on  budget.  Due  to  tight  budgets,  and 
rising  maintenance  costs,  deviations  either  above  the  line  or  below  may  lead  to  failure  to 
achieve  mission  targets.  FHE  was  also  cited  as  important  to  managers  at  higher 
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headquarters  levels,  such  as  AMC  or  ACC,  to  determine  how  the  enterprise  is  meeting 
the  established  flying  hour  goals.  This  in  turn  would  determine  how  budget  dollars  are 
disseminated. 

Metrics  connected  with  the  Budget  Management  factor  were  found  to  meet  the  criteria 
for  good  metrics  at  a  both  a  strategic  and  tactical  level.  They  allowed  senior  level 
decision  makers  and  weapon  system  managers  to  closely  monitor  the  affected  processes, 
but  also  allow  base  level  managers  to  effectively  execute  the  their  process  that  impacted 
this  factor. 

Valuation  Characteristic  Factors 

Valuation  Characteristic  factors  are  the  underlying  themes  identified  by  respondents  that 
impact  how  a  metric  was  perceived.  These  are  the  common  characteristic  found  in  this 
study  that  directly  impacted,  either  positively  or  negatively  the  value  placed  on  a  metric. 
The  distribution  of  each  Valuation  Characteristic  factor  by  metric  is  shown  in  Table  4.5. 
As  with  the  previous  themes,  the  categories  indentified  were  sorted  in  order  of  mention. 
Each  category  is  discussed  below. 


Table  4.3,  Valuation  Characteristic  Factor  Distribution 


Non-Mission  Capable  Maintenance  rate 

Repeat  rate 

Recur  rate 

Non-Mission  Capable  Supply  rate 

Avg  Age  of  ETAR 

Utilization  rate 

M1CAP  Hours 

Maint  Scheduling  Effectiveness  rate 

Fix  rate 

Non-Mission  Capable  rate 

Logistiscs  Departure  Reliablility  rate 

Avg  Delay  Discrepency 

Partially  Mission  Capable  rate 

Fully  Mission  Capable  rate 

Mission  Capable  rate 

Total  Non-Mission  Capable  Maint  rate 

Programmed  Avg  Sortie  Duration  rate 

Actual  Avg  Sortie  Duration  rate 

Total  Non-Mission  Capable  Supply  rate 

Break  rate 

Flying  Scheduling  Effectiveness  rate 
Chargeable  Deviation  rate 

Flying  Hour  Execution  rate 

Possessed  Aircraft  rate 

Cannibalization  rate 
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Echelon  Value  Factor 

Echelon  value  was  identified  15  times  by  respondents.  Echelon  value  was  identified  as 
the  characteristic  of  a  metric  having  value  at  different  levels  of  the  chain  of  command. 
This  is  where  a  metric  can  meet  the  3  goals  from  both  the  strategic  level  through  to  the 
tactical  level. 


It  was  found  to  be  a  factor  that  was  associated  with  split  evaluations  of  the  metrics.  Some 
metrics  were  identified  as  effective  across  both  tactical  and  strategic  fields  of  influence, 
such  a  Average  Delayed  Discrepancy  and  Mission  Capable  rate.  Others  were  cited  as  to 
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being  only  important  to  the  strategic  level,  such  as  Total  Non  Mission  Capable  rate  and 
Flying  Hour  Execution  rate.  Logistics  Departure  reliability  was  cited  as  relating  to  this 
factor.  One  respondent  stated  “If  I  am  General  Merchant  at  AMC  A4  because  this  all 
falls  under  my  portfolio  then  this  is  important. .  .but  to  me  at  the  field  level  it  is  not.  As  I 
don’t  have  an  aerial  port  and  don’t  control  the  fleet  service  trucks,  I  only  care  about 
maintenance  delays”.  Metrics,  such  as  MC  rate  and  ETAR,  were  also  identified  as 
playing  a  role  as  a  managers  score  card.  Senior  level  executives  placed  high  emphasis  on 
these  metrics  as  indications  of  units  perfonnance. 

Echelon  value  was  found  to  have  a  mixed  ability  in  affecting  how  metrics  achieved  the  3 
goals.  Metrics  that  displayed  echelon  value  characteristics  that  spanned  both  strategic 
and  tactical  focus,  led  to  driving  the  correct  behaviors,  provided  understanding,  and  led  to 
improvements.  However  there  are  other  metrics  that  are  only  associated  with  the 
strategic  side  of  this  echelon  pairing.  Those  metrics  need  to  be  closely  evaluated  to  see 
how  to  better  align  them  across  all  levels,  whether  they  are  tracked  and  reported  only  at 
the  strategic  level,  or  are  changed  to  be  better  aligned  across  all  levels. 

Redundancy  Value  Factor 

Redundancy  Value  was  cited  8  times  within  the  interviews.  Redundancy  Value  is  the 
factor  that  was  associated  with  metrics  that  repeated  the  same  information  as  other 
metrics.  These  metrics  were  found  to  have  little  value  by  respondents.  Redundant 
metrics  add  no  values  as  it  takes  time  and  resources  to  report  on  the  same  infonnation 
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given  by  another  source.  Many  metrics  were  identified  as  being  redundant  by 
respondents.  Abort  rate,  Partially  Mission  Capable,  Fully  Mission  Capable,  Total  Non 
Mission  Capable  M/B/S  were  all  identified  a  redundant  metrics.  These  metrics  all  had 
data  captured  and  reported  in  other  measures.  They  all  were  evaluated  low,  and  several 
specifically  identified  as  needing  to  be  removed  due  to  the  large  numbers  of  metrics  that 
respondents  are  already  having  to  answer  to. 

While  redundant  measures  provide  the  same  capabilities  as  other  metrics  for  meeting  the 
3  goals,  they  are  not  needed.  If  a  metric  already  drives  the  correct  behavior,  creates 
understanding  and  leads  to  improvement,  a  second  one  that  does  the  exact  same  thing 
should  be  considered  waste. 

Data  Reporting  Value  Factor 

Data  Reporting  Value  factors  were  cited  14  times  by  respondents  during  the  conducted 
interviews.  These  factors  were  cited  as  the  characteristics  of  metrics  that  effectively 
capture  the  correct  data  required  by  maintenance  managers. 

Data  Reporting  factors  were  found  to  rate  positively  when  they  allowed  metrics  to  trend 
historical  data,  had  clear  data  goals  that  tied  them  to  key  processes,  and  in  the  case  of  the 
new  proposed  metrics  eliminated  data  ambiguity  and  captured  all  the  data  in  the  process. 
However  in  addition  to  the  positive  factors,  respondents  also  cited  that  some  metrics, 
such  as  Mission  Capable  rate,  do  not  provide  the  right  infonnation  to  make  good 
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decisions.  While  Mission  Capable  rate  will  tell  how  much  of  the  aircraft  fleet  is  fixed,  it 
may  not  paint  the  whole  picture  of  what  is  truly  available.  It  does  not  align  effectively 
with  the  strategic  direction  of  provide  airframe  capability.  Thus  in  this  case,  data 
reporting  characteristics  resulted  in  lower  ratings  by  respondents. 

As  with  echelon  value  discussed  previously,  whether  a  metric  has  the  positive  or  negative 
aspect  of  the  factor  determines  its  effectiveness  in  helping  metrics  achieve  the  3  goals. 
Based  on  what  was  identified  from  the  interviews,  more  often  than  not  Data  Reporting 
Value  factors  will  lead  to  achieving  the  3  goals,  but  each  metric  with  this  characteristic 
will  need  to  be  evaluated  closely. 

Specificity  Value  Factor 

Specificity  Value  was  identified  9  times  by  respondents  as  characteristic  within  the 
metrics  evaluated.  Specificity  Value  is  the  ability  of  metrics  to  provide  a  specific  and 
complete  picture  of  the  processes  they  report  on.  Specificity  was  cited  for  how  metrics 
effectively  told  the  story  of  a  process. 

Metrics  associated  with  the  Specificity  Value  factor,  were  found  to  have  poor  reflection 
of  the  true  processes  or  capabilities  of  the  enterprise.  These  metrics,  such  as  Mission 
Capable  rate,  did  not  always  convey  the  true  story  of  a  process.  For  example,  an  aircraft 
could  be  reported  partially  mission  capable  or  even  fully  mission  capable  for  an 
inoperative  sub  system  by  regulation.  However,  when  given  to  an  aircrew  they  would  not 
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take  the  plane,  as  they  would  need  that  sub-system  for  the  mission  assigned.  This  could 
be  due  to  different  rule  for  operation  areas,  such  as  home  station  verses  in  CENTCOM,  or 
differences  between  what  maintenance  manager  and  operations  manager  consider 
mission  required  systems. 

Metrics  that  had  subjective  or  unclear  input  standards  were  also  cited  under  specificity. 
Additionally  metrics  that  captured  data  on  a  process  that  the  manager  could  not  effect 
were  rated  lower.  Abort  rate  is  a  metric  that  was  provided  as  an  example.  While  an 
aircraft  can  abort  for  a  maintenance  issue,  they  also  abort  for  crew  issues  such  a  pilot 
getting  sick.  Maintenance  managers  cannot  control  this  issue,  but  still  have  to  answer  to 
the  metric.  Additionally,  metrics  that  only  reported  on  part  of  a  process  were  associated 
with  this  factor.  Logistics  Departure  Reliability  reports  only  on  missions  taking  off  on 
time,  however  there  are  many  process  steps  prior  to  take  off  that  are  not  captured.  This 
fails  to  provide  managers  the  complete  picture,  and  may  lead  to  loss  opportunities  for 
improvement. 

Specificity  Value  factor  was  found  to  inhibit  metrics  to  achieve  the  three  goals  or  metrics. 
While  sometimes  these  metrics  will  drive  behavior,  by  not  showing  managers  the  total 
picture,  understanding  and  improvement  cannot  happen. 
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Poor  Criteria  Value 


Poor  Criteria  was  identified  9  times  by  respondents  as  a  characteristic  factor  of  metrics 
Metric  criteria  are  the  underlying  requirements  for  what  infonnation  is  reported  by 
metrics.  They  define  how  data  is  pulled  for  the  processes  monitored  as  well  as  how  it  is 
calculated.  Poor  Criteria  also  determine  how  well  metrics  align  with  the  strategic  intent 
for  the  maintenance  enterprise. 

Respondents  identified  metrics  with  this  factor  as  having  non-relevant  data  captured,  such 
as  with  Average  Delayed  Discrepancies,  where  information  on  future  scheduled 
maintenance  activities  are  included  in  the  count  of  open  maintenance  requests,  thus 
skewing  the  data.  Metrics  were  found  to  have  equation  components  that  appeared  to 
have  no  relevancy  to  what  was  being  measured.  One  example  provided  was  the  case  of 
Cannibalization  rate.  The  denominator  for  that  equation  is  based  on  100  sorties. 
Respondents  asked  what  the  numbers  of  sorties  has  to  do  with  the  number  of  CANNs.  A 
better  measure  might  be  per  number  of  parts  ordered,  or  per  number  of  maintenance 
actions.  Additionally,  differences  in  policy  and  regulations  impede  a  metrics  ability  to 
tell  the  true  story.  Often  conflicts  between  the  Mission  Essential  System  List  used  by 
maintenance  and  Mission  Essential  List  used  by  operations  to  status  aircraft  have 
conflicting  requirements.  This  then  leads  to  false  data  on  status  of  aircraft  to  be  reported. 

Poor  Criteria  was  found  to  have  no  positive  effect  on  metrics  ability  to  achieve  the  3 
goals.  Correct  behavior  cannot  be  developed  if  the  metrics  lead  managers  to  focus  on 
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non-critical  processes.  Additionally,  any  improvements  made  based  on  bad  criteria  may 
not  hold  value,  as  it  may  not  align  with  the  strategic  goals  of  the  enterprise. 

Poor  Decision  Value  Factor 

Poor  Decision  Value  Factors  were  identified  7  times  by  respondents  in  their  evaluation  of 
the  28  metrics.  Poor  Decision  Value  factors  are  the  characteristics  found  in  metrics  that 
lead  mangers  into  making  the  wrong  decisions  based  on  the  metrics. 

Metrics  that  lend  themselves  to  “Chasing  numbers”  were  identified  as  being  associated 
with  this  factor.  Chasing  numbers  was  defined  as  when  managers  waste  excessive 
manpower  and  resources  to  fix  aircraft  that  aren’t  need  to  meet  any  mission  requirements, 
often  Mission  Capable  rate  is  “chased”.  Often  managers  will  make  decisions  to  chase  a 
lagging  indicator  without  any  consideration  to  other  factors,  such  as  events  beyond  the 
control  of  any  maintenance  personnel  that  cannot  be  overcome  until  the  situation  resolves 
itself.  Also,  a  few  metrics  were  cited  as  being  used  for  personnel  performance  reports, 
but  not  for  any  real  process  decisions.  Again,  these  are  examples  of  attempting  to 
improve  metrics  for  reasons  other  than  improving  the  maintenance  enterprise. 

Poor  Decision  Value  factors  associated  with  metrics  do  not  allow  metric  to  meet  the  three 
goals.  While  these  metrics  may  lead  to  improvement,  it  is  often  in  the  wrong  area,  thus 
wasting  effort  and  resources.  This  metric  factor  leads  to  driving  the  wrong  behavior,  and 
potentially  clouds  understanding  of  what  is  really  going  on  in  the  observed  processes. 
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Clarity  Value  Factor 

Clarity  Value  factors  were  cited  6  times  by  respondents  in  their  evaluation  of  the  selected 
metrics.  Clarity  Value  factor  are  the  characteristics  of  metrics  that  provide  clear  and 
unambiguous  understanding.  These  factors  were  mostly  found  in  evaluations  of 
translation  and  understanding  within  the  interviews. 

Respondents  were  found  to  associate  Clarity  value  with  the  positive  aspects  of  metrics. 
Proposed  metrics  were  found  to  provide  clear  concise  definitions  over  previous  metrics. 
Thus  were  rated  high  on  the  translation  and  understanding  scale.  Additionally,  these 
factors  were  also  cited  in  proposed  metrics  that  were  almost  exactly  equivalent  to  current 
metrics,  such  as  with  Break  or  Fix  rate. 

Clarity  Value  factors  were  found  to  allow  metrics  to  achieve  all  three  goals.  Metrics 
with  these  factors  were  all  rated  high  in  their  ability  to  direct  the  correct  behaviors.  They 
provide  clear  understanding  and  control  of  processes.  Finally  these  metrics  effectively 
lead  to  improvement  within  the  maintenance  management  enterprise. 

Understanding  Value  Factor 

Understanding  factors  are  those  characteristics  that  affect  a  metric’s  ability  to  be  clearly 
understood  as  well  as  how  it  can  be  used  to  improve  processes.  Understanding  factors 
were  cited  4  times  by  respondents  during  the  interviews. 
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These  factors  were  not  only  cited  directly  by  respondents,  but  were  also  found  to  be  a 
characteristic  with  many  new  proposed  metrics.  Lack  of  understanding  was  stated  to  be 
responsible  for  metrics  being  used  ineffectively,  such  as  when  personnel  don’t  know  the 
rules  for  identifying  landing  codes  in  order  to  populate  Break  Rate  correctly.  Also  many 
lower  ratings  for  translation  and  understanding  were  attributed  to  manager’s  lack  of 
understanding  the  civilian  metric  tenninology  found  in  the  SCOR  and  Oracle  metrics. 

Understanding  factors  were  found  to  not  allow  the  metrics  they  were  associated  with  to 
meet  the  three  goals.  By  failing  to  provide  clear  understanding,  managers  could  not  use 
metrics  to  make  to  correct  process  decisions,  thus  failing  to  create  improvement  within 
their  maintenance  processes. 

Metric  Distortion  Value  Factor 

Metric  Distortion  Value  is  the  characteristic  of  a  metric  determining  how  easily  it  can  be 
intentionally  or  unintentionally  manipulated.  It  was  cited  4  times  by  respondents  in 
discussions  about  the  28  evaluated  metrics. 

Metrics  associated  with  this  factor  were  all  identified  as  having  subjective  or  unclear  data 
input  standards.  The  lack  of  clear,  defined,  concrete  data  entry  rules  can  lead 
maintenance  personnel  to  enter  metric  data  wrong.  One  example  given  was  input 
parameters  for  Break  Rate.  Break  Rate  tracks  landing  status  Code/Alpha  -3  maintenance 
issues.  However  not  every  identified  maintenance  issue  is  caught  during  debrief.  One 
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respondent  asked  “If  problem  is  found  1  hour  after  landing,  is  it  still  a  Code/Alpha  -  3 
break?”.  This  is  the  essential  problem  with  metrics  that  are  affected  by  Distortion  Value 
factors. 

Metrics  that  are  associated  with  Metric  Distortion  factors  may  seem  to  achieve  the  3 
goals  at  first  look.  However  if  the  data  captured  and  reported  is  bad,  does  the  metric 
really  direct  the  correct  behavior,  provide  understanding  or  lead  to  improvement?  Based 
on  what  was  identified  by  respondents,  it  does  not  seem  to  meet  these  goals. 

Process  Control  Value  Factor 

Process  Control  Value  factors  were  cited  by  respondents  8  times.  Process  control  is  the 
characteristic  of  metrics  that  indicates  how  much  or  how  little  control  is  given  by  the 
metric.  It  also  was  cited  in  response  to  what  portion  of  a  process  the  metrics  measured 
and  oversaw. 

Metrics  associated  with  this  factor  were  found  to  only  capture  one  or  two  steps  in  a 
process,  leaving  others  unseen  by  managers.  Logistics  Departure  Reliability  (LDR)  is 
one  of  these  metrics.  Aircraft  launches  use  schedule  of  events  (SOE)  to  drive  this 
process.  Each  SOE  can  have  over  8  separate  process  managers  involved.  The  current 
metric,  LDR,  will  only  track  the  last  step  in  that  chain,  the  actual  launch  time.  Thus 
managers  may  never  have  true  insight  into  any  problems  that  may  have  been  overcome 
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by  extraordinary  means  in  order  to  launch  the  aircraft  on  time,  because  the  metrics  mask 
the  process. 

Also  respondents  cited  that  certain  metrics  have  no  value  to  lower  echelons  as  they  are 
controlled  by  more  senior  managers  or  by  other  agencies,  such  as  with  Primary  Aircraft 
Inventory  rate  or  MICAP  rate.  They  may  give  insight  into  those  process  but  since  they 
provide  no  control  to  maintenance  managers  they  hold  little  value. 

Process  Control  factors  were  found  to  be  split  in  their  ability  to  help  metrics  achieve  the 
three  goals.  While  they  do  provide  control,  understanding  and  lead  to  improvement,  they 
often  do  this  for  agencies  other  than  the  maintenance  manager  that  has  to  report  on  these 
metrics.  These  metrics  may  need  to  be  tracked  and  monitored,  but  more  likely  not  within 
in  the  tactical  maintenance  management  level. 

Metric  Evaluation  Factor  Summary 

While  not  an  all  encompassing  list  of  themes,  the  previous  categories  discussed  give 
insight  into  how  to  evaluate  metrics  in  tenns  of  the  three  goals  of  metrics:  To  drive 
behavior,  to  create  understanding  and  to  lead  to  improvement.  These  themes  identified 
can  be  used  to  take  an  in-depth  look  at  metrics  that  are  being  considered  for  inclusion  into 
management  systems,  like  ECSS.  Being  able  to  evaluate  a  metric’s  association  with 
these  themes,  it  may  assist  in  detennining  its  suitability  for  inclusion  into  a  management 
program.  Table  4.4  provides  a  summary  distribution  by  category  and  goal  impact. 
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Table  4.4  Summary  of  Theme  Distribution  by  Goal  Impact 
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ability  to  meet  3  goals  of  metrics 

If  metrics  are  associated  with  a  category  on  the  left  side  of  the  table  that  achieves  the 
three  goals  then  it  may  be  considered  for  inclusion  with  little  modification.  Metrics 
associated  with  the  middle  category  of  Partial  ability,  should  be  evaluated  to  how  these 
factors  impact  metrics.  If  the  impact  is  positive  then  the  metrics  can  be  considered  for 
inclusion,  but  if  negative  then  serious  re-evaluation  may  be  needed.  If  the  metric 
associates  with  the  right  side  of  the  table  then  serious  evaluation  of  the  metric  should  be 
accomplished  in  order  to  better  align  it  with  the  three  goals  of  metrics  before  inclusion 
into  a  management  system. 
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Cross  Metric  Analysis  of  Translation  and  Understanding 

After  completing  the  comparison  framework  with  the  28  metrics,  the  study  found  eight 
Level  3  metric  comparisons  from  current  metrics  to  the  SCOR  list.  Eight  Level  2 
comparisons  were  established  between  current  maintenance  metrics  and  the  SCOR  list. 
Four  current  maintenance  metrics  were  found  as  Level  One  comparisons  with  the  SCOR 
list  metrics.  Nine  metrics  did  not  have  any  counterpart  metrics  in  the  SCOR  metrics  list. 

In  terms  of  SCOR  list  metric  compared  to  the  key  perfonnance  measures  outlined  by  the 
Oracle  software  manuals,  none  of  the  18  SCOR  metrics  translated  to  Oracle.  The  study 
found  that  2  metrics  actually  matched  from  current  Air  Force  metrics  directly  to  the 
Oracle  software,  but  these  metrics  had  no  counterpart  in  the  SCOR  metric  list.  The 
remaining  6  of  the  air  force  metrics  were  found  to  have  no  viable  comparison  metrics. 
The  results  of  this  analysis  are  summarized  in  Table  4.4,  showing  each  connection  across 
the  three  models. 
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Table  4.5.  Summary  Comparison  Framework 


Each  metric  set  was  evaluated  by  maintenance  management  experts  using  a  5  point  Likert 
rating  scale.  Each  interview  rating  was  tabulated  and  summed  up  across  all  twelve 
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interviews  to  produce  a  total  composite  score  for  Translation  and  Understanding  of  the 
proposed  metrics  as  shown  in  Table  4.5. 

Table  4.6.  Translation/Understanding  Evaluation  Results 
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Scores  had  a  possible  range  of  2  points  to  10  points  for  individual  ratings,  and  a  range  of 
24  points  to  120  points  possible  for  the  aggregated  total  across  all  interviews.  In  the 
Translation/Understanding  categories,  metric  total  score  values  ranged  from  82  points  on 
the  bottom  to  a  maximum  of  1 13  points.  The  lowest  scored  metric  Average  Delayed 
Discrepancies  on  the  composite  table  was  still  relatively  high  in  value  in  terms  of  the 
absolute  scoring  range,  indicating  that  all  evaluated  metrics  were  found  to  be  adequately 
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translated  and  were  understandable.  This  shows  that  the  developed  method  of  translating 
metrics  based  on  key  attribute  pattern  matching  was  valid  and  produced  satisfactory 
results.  Eight  metric  sets  did  not  have  any  Translation/Understanding  scores  because 
they  either  did  not  have  translations  into  the  new  system  or  were  new  metrics  proposed  to 
the  SME. 


The  compiled  metrics  table  was  then  cross  referenced  to  the  comparison  framework  in 
Table  4. 1  to  determine  if  level  of  comparison  had  any  relation  to  rating  score.  The  study 
found  that  Level  3  translations  scored  highest  within  the  table,  while  Level  2  and  Level  1 
translation  sets  comprised  the  lower  half  of  the  table.  While  some  of  the  Level  1  metrics 
did  score  higher  than  most  Level  2,  this  was  attributed  to  the  clear  and  precise  definitions 
provided  in  the  SCOR  definitions.  The  metrics  that  were  scored  the  lowest  were  the 
metrics  that  had  appeared  to  have  the  biggest  difference  in  terminology  between  Air 
Lorce  and  SCOR  model  metrics. 

Cross  Metric  Analysis  for  Importance  and  Effectiveness 

In  order  to  determine  which  of  the  Air  Lorce  aircraft  maintenance  metrics  evaluated 
should  be  transitioned  into  ECSS,  each  metric  was  evaluated  by  subject  matter  experts  on 
the  question  of  importance  and  effectiveness  of  the  metric.  Each  metric  analysis  was 
evaluated  against  the  Metrics  Effectiveness  Checklist  found  in  Chapter  2  (Table  2.3) 
based  on  the  interview  data  provided  in  addition  to  the  Likert  scale  rating.  Each 
interview  rating  was  tabulated  and  summed  up  across  all  twelve  interviews  to  produce  a 
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total  composite  score  for  Translation  and  Understanding  of  the  proposed  metrics  as 
shown  in  Table  4.6. 


Scores  had  a  possible  range  of  2  points  to  10  points  for  individual  ratings,  and  a  range  of 
24  points  to  120  points  possible  for  the  aggregate  score.  In  the  Importance/Effectiveness 
categories  metric  total  score  values  ranged  from  60  points  on  the  low  end  to  a  maximum 
of  120  points. 
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The  table  appears  to  have  two  distinct  separation  points  among  the  metrics.  These  points 
divide  the  metrics  into  three  groups  with  the  high  group  falling  between  102  to  90  points, 
the  moderate  group  between  87  to  80  points  and  the  low  group  composite  scores  between 
79  to  60.  The  table  also  shows  that  most  of  the  metrics  had  composite  scores  that 
gravitated  towards  the  higher  end  of  the  point  scale.  This  indicates  that  while  the  metrics 
are  clearly  ordered  in  importance  and  effectiveness  within  the  table,  the  SME’s  hold  most 
of  the  metrics  as  important  and  effective  in  terms  of  the  absolute  scale  of  the  evaluations. 

Each  metric  summary  was  then  applied  to  the  adapted  Metrics  Effectiveness  Checklist 
(Table  2.3).  Each  metric  evaluation  was  subjected  to  the  series  of  15  yes/no  questions 
based  on  the  information  provided  by  the  field  interview  analysis.  The  results  are 
displayed  in  Table  4.7. 
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Table  4.8  Metrics  Effectiveness  checklist  evaluation  results. 
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11 

4 

Avg  Delay  Discrepency  to  Maintenance  Request  Backlog 

11 

4 

Mission  Capable  to  Weapon  System  Availability 

11 

4 

Flying  Scheduling  Effectiveness/Chargeable  Deviation  rate  to  Plan  Changes  within  Execution  Window 

11 

El 

Non-Mission  Capable  Rate  to  Not  Mission  Capable  Both 

10 

5 

2 

Programmed  Avg  Sortie  Duration 

10 

m 

n 

Break  rate  to  Break  rate 

10 

5 

2 

Actual  Avg  Sortie  Duration 

9 

■ 

ME 

Utilization  Rate  to  Utilization  Rate 

9 

m 

WM 

Logistiscs  Departure  Reliablility  Rate  to  On-time  Operations  Starts* 

9 

m 

■E 

Flying  Hour  Execution  Rate  to  Operational  Programmed  Time  Accuracy 

8 

m 

Possessed  Aircraft  Rate  to  Depot  Possesed  Percentage 

8 

m 

Primary  Aircraft  Inventory  to  Total  Active  Inventory 

8 

m 

Fully  Mission  Capable  Rate 

7 

m 

Total  Non-Mission  Capable  Supply  Rate  to  Total  Not  Mission  Capable  Supply  Rate 

7 

8 

0.88 

Cannibalization  Rate  to  Canibalization  Rate 

7 

m 

Total  Non-Mission  Capable  Maint  Rate  to  Total  Not  Mission  Capable  Maint  Rate 

5 

Total  Non-Mission  Capable  Rate 

5 

Abort  rate 

4 

EB 

This  study  determined  that,  based  on  the  ratio  of  yes  to  no  responses,  1 1  metrics  had  a 
ratio  greater  than  2.  When  compared  with  Table  4.6,  7  had  high  evaluation  scores,  3  had 
moderate  evaluation  scores,  and  1  metric  had  a  low  evaluation  score.  Nine  Metrics  had 
ratios  between  1  and  2.  When  compared  with  Table  4.6,  3  had  high  evaluation  scores  and 
6  had  low  evaluation  scores.  Six  Metrics  were  found  to  have  ratios  less  than  1.  When 
compared  with  Table  4.6,  2  had  moderate  evaluation  scores  and  4  were  had  low 
evaluation  scores. 
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The  study  found  that  the  metric  did  trend  together  on  both  the  numerical  evaluations  and 
the  checklist  evaluations.  Discrepancies  were  noted  however,  but  are  believed  to  relate  to 
the  response  rates  of  the  evaluation  questions.  All  respondents  provided  numerical 
ratings  of  every  metric,  but  not  all  provided  supporting  discussion  as  to  the  supporting 
factors  of  the  ratings.  Therefore  those  supporting  comments  could  not  be  factored  into 
the  checklist  evaluation.  However,  based  on  what  was  extracted  from  both  the  numerical 
and  qualitative  evaluations  given,  the  metrics  checklist  in  conjunction  with  expert 
evaluation  can  be  utilized  as  an  effective  tool  for  evaluating  the  metrics  importance  and 
effectiveness. 

Summary 

This  chapter  presented  the  analysis  results  of  the  translation  and  empirical  study  of  the  28 
selected  aircraft  maintenance  metrics.  The  development  of  the  translation  framework  and 
results  of  the  structured  interviews  were  presented.  Analysis  of  the  metric  characteristics 
was  discussed.  Finally,  cross  case  analysis  of  the  metric  evaluations  was  performed  in 
order  to  determine  the  effectiveness  of  the  comparison  framework  and  to  provide 
recommendations  for  metric  disposition. 
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V.  Conclusions 


Overview 

This  final  chapter  will  summarize  and  present  the  conclusions  of  this  study.  It  will 
present  the  final  results  and  recommendations  of  the  two  research  questions.  In  addition 
discussion  of  the  assumptions  and  limitations  will  be  presented  along  with  the 
significance  of  the  research  and  recommendations  for  future  studies. 

As  organizations  evolve  and  grow  new  and  better  ways  must  be  developed  to  meet  future 
challenges.  One  of  these  ways  is  the  implementation  of  an  Enterprise  Resources 
Planning  (ERP)  program.  For  the  US  Air  Force,  The  Expeditionary  Combat  Support 
System  will  be  one  cornerstone  of  Logistics  Transfonnation.  The  establishment  of  this 
program  will  allow  future  logisticians  to  fully  understand  and  efficiently  manage  any  and 
all  logistics  processes. 

An  ERP  implementation  endeavor  consists  of  a  large  number  of  critical  areas  that  have  to 
be  addressed.  Management  Support,  Business  Process  Reengineering,  Strategy  and 
Governance  modeling,  and  Legacy  Systems  Evaluation  and  Conversion  are  just  a  few  of 
the  key  areas  that  need  to  be  managed  successfully  for  effective  ERP  implementation. 
This  study  has  focused  on  one  area,  Legacy  Systems  Evaluation  and  Conversion, 
specifically  the  integration  of  the  Air  Force  Aircraft  Maintenance  Metrics  program  into 
ECSS. 
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Research  Conclusions 


The  purpose  of  this  study  was  to  develop  a  method  to  translate  current  Air  Force  Metrics 
into  the  ECSS  program.  In  order  to  do  this  a  comparison  framework  was  developed  and 
validated  through  structured  field  interview  evaluations  of  maintenance  management 
professionals. 

Research  Question  One 

How  well  do  current  metrics  program  measures  translate  into  the  Expeditionary 
Combat  Support  System  Enterprise  Resource  Planning  framework? 

This  question  was  answered  in  two  parts.  First  a  theoretical  framework  was  developed 
based  on  the  key  metric  attributes:  Nomenclature,  Definition  and  Equation.  Once  the 
selected  units  of  analysis  were  fit  within  the  framework,  the  metric  translation  sets  were 
presented  to  subject  matter  experts  in  the  field  of  aircraft  maintenance  management  in 
order  to  determine  the  validity  and  success  of  the  translations. 

The  study  has  concluded  that  using  pattern  matching  analysis  of  three  key  attributes, 
Nomenclature,  Definition  and  Equations,  current  Air  Force  metrics  can  be  translated  to 
the  metrics  identified  by  the  Logistics  Transfonnation  Office  (LTO)  in  the  Supply  Chain 
Organization  Reference  (SCOR)  Model  and  then  to  the  Oracle  ERP  software  Suite. 
However  the  study  found  that  while  17  of  the  26  current  metrics  translated  into  the  SCOR 
model,  they  did  not  have  any  translations  into  Oracle.  In  order  to  convert  these  metrics 
the  LTO  will  have  to  build  them  into  the  software  package.  3  metrics  did  translate  into 
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Oracle  and  can  be  incorporated  directly  into  ECSS.  Ultimately,  23  metrics  will  have  to 
be  built  into  the  Oracle  software  platform. 

Research  Question  Two 

Which  maintenance  metrics  should  be  translated  into  the  new  enterprise  resource 
planning  framework? 

Metrics,  as  discussed  in  Chapter  2,  ultimately  need  to  serve  three  purposes.  Metrics 
must: 

1 .  Correctly  drive  decision  behavior  and  provide  control 

2.  Communicate  the  process  clearly  and  create  understanding 

3.  Be  able  to  lead  to  improvement  within  the  process 

Each  of  the  28  metrics  was  evaluated  for  importance  and  effectiveness  in  order  to 
detennine  their  suitability  for  translation  into  ECSS.  Ten  metrics  were  found  to  be 
suitable  for  incorporation  into  ECSS.  These  metrics  were  able  to  have  at  least  1 1  yes 
answers  on  the  Metrics  Effectiveness  Checklist,  resulting  in  yes/no  ratios  of  at  least  2.75. 

Ten  Metrics  need  to  be  re-evaluated  by  senior  maintenance  mangers  in  order  to  better 
align  them  with  the  strategic  intent  of  the  maintenance  metrics  program.  While  these 
metrics  did  have  more  yes  answers  than  no,  they  will  require  further  analysis  in  order  to 
detennine  how  to  improve  their  effectiveness  for  maintenance  management. 
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Finally  six  metrics  were  recommended  for  removal  from  consideration.  These  metrics  all 
had  yes/no  ratios  of  less  than  one.  As  indicated  by  the  analysis  these  metrics  are  either 
redundant  or  have  little  value  to  maintenance  managers.  Table  5.1  shows  the  results 
break  down  for  each  metric. 

Table  5.1  Metric  Recommendations 


Avg  Age  of  ETAR 

Incorporate 

MICAP  Hours 

Incorporate 

Partially  Mission  Capable  Rate 

Incorporate 

Non-Mission  Capable  Rate  Supply  to  Not  Mission  Capable  Supply 

Incorporate 

Maint  Scheduling  Effectiveness  to  Attainment  to  Plan 

Incorporate 

Fix  rate  to  Fix  rate 

Incorporate 

Non-Mission  Capable  Rate  Maint  to  Not  Mission  Capable  Maint 

Incorporate 

Repeat/Recur  rate  to  First  time  Fix  rate 

Incorporate 

Avg  Delay  Discrepency  to  Maintenance  Request  Backlog 

Incorporate 

Mission  Capable  to  Weapon  System  Availability 

Incorporate 

Flying  Scheduling  Effectiveness/Chargeable  Deviation  rate  to  Plan  Changes  within 

Execution  Window 

Re-evaluate 

Non-Mission  Capable  Rate  to  Not  Mission  Capable  Both 

Re-evaluate 

Programmed  Avg  Sortie  Duration 

Re-evaluate 

Break  rate  to  Break  rate 

Re-evaluate 

Actual  Avg  Sortie  Duration 

Re-evaluate 

Utilization  Rate  to  Utilization  Rate 

Re-evaluate 

Logistiscs  Departure  Reliablility  Rate  to  On-time  Operations  Starts* 

Re-evaluate 

Flying  Hour  Execution  Rate  to  Operational  Programmed  Time  Accuracy 

Re-evaluate 

Possessed  Aircraft  Rate  to  Depot  Possesed  Percentage 

Re-evaluate 

Primary  Aircraft  Inventory  to  Total  Active  Inventory 

Re-evaluate 

Fully  Mission  Capable  Rate 

Remove 

Total  Non-Mission  Capable  Supply  Rate  to  Total  Not  Mission  Capable  Supply  Rate 

Remove 

Cannibalization  Rate  to  Canibalization  Rate 

Remove 

Total  Non-Mission  Capable  Maint  Rate  to  Total  Not  Mission  Capable  Maint  Rate 

Remove 

Total  Non-Mission  Capable  Rate 

Remove 

Abort  rate 

Remove 

Assumptions/Limitations 

Several  assumptions  were  made  in  the  scope  of  this  research.  1)  The  established  Air 
Force  metrics  program  will  be  used  to  baseline  the  transition  into  ECSS.  2)  The  data 
collected  is  to  be  considered  accurate  and  valid  and  the  interviewed  experts  have  depth 
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and  breadth  of  experience  in  aircraft  maintenance  management.  3)  The  Oracle  software 
evaluated  will  be  the  program  implemented  by  the  ECSS  transition  team. 

Additionally  several  limitations  of  this  study  were  identified.  1)  This  is  an  exploratory 
study,  so  only  12  subjects  were  interviewed,  therefore  results  may  not  be  attributable 
across  the  entire  maintenance  community.  2)  The  potential  for  bias  on  behalf  of  the 
researcher  may  be  a  factor  as  experience  with  the  subject  matter  may  impact  the  analysis. 
3)  It  was  also  to  be  understood  that  not  all  information  pertaining  to  this  study  may  have 
been  revealed  during  the  course  of  this  research. 

Additional  Findings 

This  study  found  in  addition  to  the  answers  to  the  original  research  questions,  several 
other  findings  were  identified  that  did  not  fit  within  the  scope  of  this  research.  During 
the  interview  evaluations  of  the  Air  Force  Maintenance  metrics,  over  half  of  the 
respondents  described  a  need  for  better  personnel  management  metrics.  The  air  force 
currently  has  one  metric  that  tracks  authorized  manning  verses  assigned.  Maintenance 
managers  described  a  need  for  metrics  that  will  track  and  project  actual  manning 
capability. 

While  units  may  have  85  to  95  percent  of  authorized  personnel,  the  true  availability  of 
those  maintainers  is  not  known  or  tracked.  Personnel  are  often  out  of  pocket  for 
leadership  schools,  duties  unrelated  to  aircraft  maintenance,  or  just  out  sick.  Due  to  these 
factors,  maintenance  managers  often  cite  this  lack  of  people  as  constraints  in 
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accomplishing  tasks,  but  cannot  truly  quantify  this  pain.  By  creating  such  a  metric  a 
better  understanding  of  true  enterprise  capability  may  me  realized,  thus  providing  an 
accurate  picture  of  inherent  aircraft  capability. 

Significance  of  Research 

The  United  Stated  Air  Force  has  invested  hundreds  of  personnel  hours,  and  millions  of 
dollars  in  developing  new  logistics  management  systems.  As  discussed  earlier,  past 
logistics  operation  policies  are  no  longer  sufficient  to  meet  the  needs  of  the  war-fighter. 
Shrinking  budgets,  aging  equipment,  and  the  austere,  disparate  operating  location 
demand  sweeping  changes  in  how  the  logistic  machine  operates.  The  Expeditionary 
Combat  Support  System  (ECSS)  is  the  Air  Force  wide  Enterprise  Resource  Planning 
(ERP)  system  designed  to  tackle  these  very  challenges. 

In  order  to  effectively  create  and  field  this  system,  current  management  programs  and 
processes  must  be  evaluated  and  reengineered  if  necessary.  In  meeting  this  directive,  this 
study  has  endeavored  to  develop  a  framework  for  the  successful  translation  of  current  Air 
Force  maintenance  metrics  into  the  ECSS  system.  Successful  execution  of  this  study  will 
provide  recommendations  to  the  Logistics  Transfonnation  Office  for  disposition  of  the 
selected  metrics.  By  effectively  transitioning  from  the  old  metric  program  to  the  new, 
one  section  of  the  ECSS  program  will  have  been  accomplished.  Additionally,  it  is  hoped 
this  study  will  provide  the  tools  that  can  be  used  for  evaluating  other  Logistics  metrics 
that  are  expected  to  be  translated  into  the  ECSS  program. 
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Recommendations  for  Future  Research 


The  study  has  developed  an  effective  framework  for  evaluating  the  translation  of  current 
Air  Force  maintenance  metrics  into  ECSS.  However  this  is  not  the  complete  list  of 
metrics  in  use  by  the  logistics  community.  Future  studies  should  be  undertaken  to 
address  these  other  Air  Force  metrics.  This  will  allow  further  validation  of  the 
framework  and  will  provide  the  Logistics  Transformation  Office  a  complete  evaluation  of 
Air  Force  metrics. 

Another  recommendation  for  further  studies  is  to  expand  this  case  study  into  a  statistical 
survey  by  a  larger  population  of  maintenance  experts.  As  this  was  an  exploratory  study, 
the  researchers  selected  a  small  pool  of  interview  subjects.  Further  revelations  may  be 
developed  by  expanding  the  expert  pool  to  include  maintenance  managers  from  other 
organizations,  such  as  lighter  or  special  operations  units  was  well  as  evaluations  from 
managers  at  higher-level  headquarters. 

The  results  of  this  study  have  found  a  number  of  metrics  that  need  re-evaluation  if  no  re¬ 
engineering.  Further  exploration  should  be  undertaken  in  order  to  detennine  how  to  best 
change  these  metrics  to  align  better  with  the  strategic  goals  of  the  Air  Force  and  well  as 
the  goals  for  effective  metrics. 
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Appendix  A  Interview  Guide 


In  this  interview  you  will  be  asked  to  evaluate  the  translation  from  the  current  metrics  as 
outlined  in  Air  Force  Instruction  21-101  Aircraft  and  Equipment  Maintenance 
Management  to  the  new  metrics  incorporated  into  the  Oracle  Business  Initiative  Software 
as  part  of  the  Expeditionary  Combat  Support  System  program. 

Each  traditional  logistics  measure  is  provided  with  its  new  translated  metric,  please 
evaluate  based  on  the  provide  criteria  and  your  experience  with  maintenance  metrics. 

28  aircraft  maintenance  metrics  are  outlined  on  the  following  pages 

The  current  metric,  proposed  metric,  and  definitions  of  each  are  shown  at  the  top  of  each 

page 

There  are  4  evaluative  questions  to  answer  about  each  metric 

During  the  interview  we  will  rate  the  metrics  on  the  provided  5  point  scale  and  discuss 
the  selected  answers. 

In  additions  to  the  metric  evaluations  there  will  be  additional  supporting  questions  about 
this  study. 


Thank  you  for  your  time  and  support  in  completing  this  study. 
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Metric  1 


Current 


Proposed 


Missions  Capable  Rate 

Percentage  of  hours  aircraft  are 
mission  capable,  has  derivatives  of 
Partial,  Non,  Total  Non,  as  well  as 
sub  categories  of  Due  to 
Maintenance  or  Due  to  Supply 


(FMC  Hours  +  PMC 
Hours)/Possessed  hours)  x  100 


Weapon  System  Availability 

Percentage  of  hours  a  reported  unit 
possessed  weapon  system  was 
capable  of  performing  any  assigned 
mission 

(MC  hours/Total  Active  Inventory 
hours)  x  1 00 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 
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Metric  3 


Current 
Fix  Rate 


Proposed 
Fix  Rate 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  4 


Current 


Proposed 


Average  Delayed  Discrepancy  Rate  Maintenance  Request  Backlog 


Average  Number  of  deferred 

Number  of  open  maintenance 

maintenance  actions 

requests  on  selected  date 

(Total  (snapshot)  maintenance 

(Snapshot  of  number  of  open 

actions/ Average  Aircraft  Possessed) 

maintenance  requests) 

How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


l 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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Metric  5 


Current 


Repeat  Rate 


Percentage  of  maintenance 
discrepancies  that  occur  again  after 
next  sortie  attempt  after  being  fixed 

(Total  Repeats/Total  Reported 


Recur  Rate 

Percentage  of  maintenance 
discrepancies  that  occur  the  2nd  thru 
4th  sortie  attempts  after  being  fixed 

(Total  Recurs/Total  Reported 
Discrepancies)  x  100 


Proposed 

First  Time  Fix  Rate 


Percentage  of  maintenance  requests 
fixed  at  the  first  visit 

(Total  First  Time  Fixes/Total 
Maintenance  Requests)  x  100 


How  important  is  the  1st  current  metric  for  managing  aircraft  maintenance? 

Is  the  1st  current  metric  effective  in  managing  aircraft  maintenance? 
How  important  is  the  2nd  current  metric  for  managing  aircraft  maintenance? 
Is  the  2nd  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  6 


Current 


Proposed 


Abort  Rate  no  new  metric 

Percentage  of  missions  that  end 
prematurely  and  must  be  re- 
accomplished 

(Air  +  Ground  Aborts/Total  Sorties 
Flown  +  Ground  Aborts)  x  1 00 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 


Is  the  current  metric  effective  in  managing  aircraft  maintenance? 


1  2 

3 

4  5 

Not  Important 

Somewhat 

Very  Important 

1  2 

3 

4  5 

Not  Effective 

Somewhat 

Very  Effective 
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Metric  7 


Current  Proposed 

Maintenance  Scheduling  Effectiveness  Rate  Attainment  to  Plan 


*• 


Measures  how  well  the  execution  are 
meeting  their  respective-planned 
actions 

(#  Actual  Events/#  Planned 
Events)*  100 


Percentage  of  scheduled  maintenance 
actions  verses  accomplished 
scheduled  maintenance  actions  based 
on  assigned  points  be  maintenance 
action 

(Total  points  eamed/total  points 
assigned)  x  100 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
s  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  8 


Current 


Proposed 


CANN  Rate  CANN  Rate 


Percentage  of  cannibalization  actions 

Average  number  of  CANN  actions 

to  replace  parts  on  other  aircraft 

per  1 00  sorties  flown  for  flying 

- ► 

assets 

(Number  of  CANNs/Total  Sorties 

Flown)  x  100 

[(Number  of  Aircraft-to- Aircraft 

CANNs)  +  (Number  of  Engine-to- 

Aircraft  CANNs)  /  Total  Sorties 

Flown]  x  100 

How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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Metric  9 


Proposed 
No  new  Metric 


Percentage  of  hours  aircraft  can 
perform  some  but  not  all  assigned 
missions,  due  to  Supply, 
Maintenance  or  Both 

(PMC  Hours/Total  Possessed  Hours) 
x  100 


Current 

Partially  Mission  Capable  Rate 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 


1  2 

3 

4  5 

Not  Important 

Somewhat 

Very  Important 

Is  the  current  metric  effective  in  managing  aircraft  maintenance? 


1  2 

3 

4  5 

Not  Effective 

Somewhat 

Very  Effective 

Metric  10 


Current 


Proposed 


Non  Mission  Capable  Rate 


Not  Mission  Capable  -  Both 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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Metric  1 1 


Current 


Proposed 


NMC  -  Supply 


NMC  -  Supply 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  12 


Current 


Proposed 


NMC  -  Maintenance 


NMC  -  Maintenance 


Percentage  of  hours  aircraft  cannot 

Percentage  of  time  a  reported  unit 

perform  any  assigned  missions  due 

possessed  weapon  system  cannot 

to  maintenance 

perform  any  assigned  mission  due  to 

w 

only  maintenance 

(NMCM  Hours/Total  Possessed 

Hours)  x  100 

(NMCM  Hours/Total  Possessed 

Hours)  x  100 

How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


l 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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Metric  13 


Current  Proposed 

Total  Non  Mission  Capable  Rate  No  new  Metric 

Percentage  of  total  NMC  hours 

(Total  NMC  Hours/Total  Possessed 
Hours)  x  1 00 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  14 


Current 


Proposed 


TNMC  -  Maintenance  TNMC  -  Maintenance 


Percentage  of  total  NMC  hours  due 

Percentage  of  time  a  reported  unit 

to  Maintenance 

possessed  weapon  system  cannot 

- ► 

perform  any  assigned  mission  due  to 

(Total  NMCM  Hours/Total 

maintenance  or  both  supply  and 

Possessed  Hours)  x  100 

maintenance 

(Sum  of  NMCM  Hours  +  Sum  of 

NMCB  Hours/Total  Possessed 

Hours)  x  100 

How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


l 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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Metric  15 


Current 


Proposed 


TNMC  -  Supply  TNMC  -  Supply 


Percentage  of  total  NMC  hours  due 

Percentage  of  time  a  reported  unit 

to  Supply 

possessed  weapon  system  cannot 

- ► 

perform  any  assigned  mission  due  to 

(Total  NMCS  Hours/Total  Possessed 

supply  or  both  supply  and 

Hours)  x  1 00 

maintenance 

(Sum  of  NMCs  Hours  +  Sum  of 

NMCB  Hours/Total  Possessed 

Hours)  x  100 

How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


l 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  16 


Current  Proposed 


Flying  Hour  Execution  Rate  Operational  Programmed  Time  Accuracy 
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Metric  17 


Current  Proposed 

Flying  Scheduling  Effectiveness  Rate  Planning  Changes  within  Execution  Window 


Metric  18 


Proposed 

No  new  Metric 

Percentage  of  hours  aircraft  can 
perform  all  assigned  missions 

(FMC  Hours/Total  Possessed  Hours) 
x  100 


Current 

Fully  Mission  Capable  Rate 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 


1  2 

3 

4  5 

Not  Important 

Somewhat 

Very  Important 

1  2 

3 

4  5 

Not  Effective 

Somewhat 

Very  Effective 
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Metric  19 


Current 

Primary  Aircraft  Inventory 

Assigned  number  of  aircraft  by  MDS 
per  designated  organization 

Snapshot  of  number  of  aircraft 
assigned 


Proposed 


Total  Active  Inventory 


Inventory  assets  with  Possession 
Purpose  Code  =  CA,  CB,  CC,  CF, 
EH,  El,  IF,  PJ,  PL,  PR,  TF,  TJ,  ZA, 
ZB 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  20 


Current 


Proposed 


Logistics  Departure  Reliability  Rate  On  time  Operations  Starts 

*• 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Percentage  of  on-time  aircraft 
departures  due  to  logistics 

(Number  of  Departures-Number  of 
Logistics  Delays/Number  of 
departures)  x  100 


Percent  of  Operations  started  on  time 

(Number  of  Operations  on  time 
starts/Number  of  Operations)  x  1 00 
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Metric  2 1 


Current 


Proposed 


Possessed  Aircraft  Rate  Depot  Possessed  Percent 

*• 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Percentage  of  the  total  active 
inventory  hours  that  are  in  depot 
possessed  status 

(Sum  of  Possessed  hours/Sum  of 
TAI  hours)*  100 


Aircraft  under  control  of  owning 
designated  organization 

Snapshot  of  number  of  aircraft 
controlled  by  organization 


Metric  22 


Proposed 

no  new  metric 

Average  Sortie  length  scheduled 

(Number  of  Scheduled  Sorties/Total 
Scheduled  Sortie  Hours) 


Current 

Programmed  Average  Sortie  Duration 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 


1  2 

Not  Important 

3 

Somewhat 

4  5 

Very  Important 

1  2 

Not  Effective 

3 

Somewhat 

4  5 

Very  Effective 
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Metric  23 


Current  Proposed 

Actual  Average  Sortie  Duration  no  new  metric 

Average  Sortie  length  executed 

(Number  of  Sorties  executed/Total 
executed  Sortie  Hours) 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 


1  2 

Not  Important 

3 

Somewhat 

4  5 

Very  Important 

1  2 

Not  Effective 

3 

Somewhat 

4  5 

Very  Effective 

Metric  24 


Current 


Proposed 


Utilization  Rate 


Utilization  Rate 


How  important  is  the  current  metric  for  managing  aircraft  maintenance? 
Is  the  current  metric  effective  in  managing  aircraft  maintenance? 
Does  the  proposed  metric  adequately  replace  the  current  metric? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Strongly 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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Metric  25 


Current 


Proposed 


Average  Age  of  ETAR 


Average  by  Day  of  open  requests  for 
engineering 

(Total  number  of  days  for  open 
ETARs/Total  number  of  ETARs) 


How  important  is  the  proposed  metric  for  managing  aircraft  maintenance? 
Would  the  proposed  metric  be  effective  in  managing  aircraft  maintenance? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 

Metric  26 


Current  Proposed 

MICAP  Hours 


Total  MICAP  Hours  attributed  to 
parts  delay 

(Total  MICAP  Hours  per  Month) 


How  important  is  the  proposed  metric  for  managing  aircraft  maintenance? 
Would  the  proposed  metric  be  effective  in  managing  aircraft  maintenance? 

Is  this  new  metric  easily  understood? 


1 

Not  Important 

2 

3 

Somewhat 

4 

5 

Very  Important 

1 

Not  Effective 

2 

3 

Somewhat 

4 

5 

Very  Effective 

1 

Poorly 

2 

3 

Somewhat 

4 

5 

Clearly 
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What  is  your  understanding  of  the  Expeditionary  Combat  Support  System  program 
(ECSS)  and  its  effect  on  the  aircraft  maintenance  community? 


What  are  the  metrics  you  would  select  a  most  important  if  you  could  only  have  5? 


What  are  the  metrics  you  would  select  if  you  had  to  eliminate  5? 


Are  there  any  metrics  not  included  in  this  list  that  you  deem  necessary  to  managing 
aircraft  maintenance? 


What  areas  of  concerns  or  fears  do  you  perceive  in  moving  from  the  current  metrics 
program  to  the  new  ECSS  system? 
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Appendix  B  SCOR  Metrics  List 


Metric 

ID 

Metric  Name 

Definition 

Supp  lemental  Definition 

Business  Rules/Formula 

Frequency 

Measuremen 

tUnit 

181 

Number  of  Planning 

changes  within 

Execution  Window 

The  number  of  times  the  plan  had 

to  be  re-ran,  or  changed  within 

the  Execution  Window.  Desired 

trend  is  minimum  number  of 

times 

N/A 

N/A 

Daily 

Percentage 

182 

Operational 

Programmed  Time 

Accuracy  (e.g.  Flying 

Hours) 

Compares  future  Operational 

Time  (i.e.  Actual  FHs)  to  actual 

Operational  Time  (i.e.  Actual 

FHs  flown)  by  location  or  group. 

Can  be  used  to  calculate  the 

accuracy  of  EOH  and  PDM 

schedules 

(*  NOTE:  We  need  to  send  this 

metric  to  M/R  *) 

N/A 

Monthly 

Percentage 

190 

Attainment  to  Plan 

Attainment  to  Plan  measures  how 

well  the  execution  organizations 

(Source,  MRO  site, 

Deliver/Return)  are  meeting  their 

respective-planned  dates  and 

quantities  in  support  of  the 

enterprise  TPMP. 

As  attainment  to  plan  increases, 

the  supply  plans  are  more 

reliable,  and  less  safety  stock  is 

required  to  cover  supply 

variability.  Can  measure  Source 

Plans,  Production  to  MPS  Plans, 

and  Distribution  Plans. 

(Actual  Event/Planned  Event)*  100 

Daily 

Percentage 

192 

Cannibalization 

(CANN)  Rates 

The  CANN  rate  is  the  average 

number  of  CANN  actions  per  100 

sorties  flown  for  flying  assets. 

Can  also  have  a  CANN  rate  for 

non-fly  assets. 

A  CANN  action  is  the  removal 

of  a  serviceable  part  from  an 

aircraft  or  engine  to  replace  an 

unserviceable  part  on  another 

aircraft  or  engine,  or  removal  of 

a  serviceable  part  to  put  into  a 

readiness  spares  package  for 

deployments. 

CANN  Rate  =  [(Number  of  Aircraft-to-Aircraft 

CANNs)  +  (Number  of  Engine-to-Aircraft  CANNs)  / 

Total  Sorties  Flown]  x  100 

Daily 

Percentage 

680 

Average  age  of 

ETARs 

Average  by  day  of  open  requests 

for  engineering  assistance 

Total  number  of  days  for  open  ETARs  divided  by 

total  of  open  ETARs 

Semi- 

Annualy 

Day 

729 

Break  Rate  (Leading 

Indicator)  -  landing 

status  codes  are  not 

finalized;  mx  and  ops 

need  to  handshake 

The  break  rate  is  the  percentage 

of  sorties  that  land  in  a  Code  3 

status.  Several  indicators  that 

follow  break  rate  are  MC, 

TNMCS,  CANN,  and  R/R. 

See  definition  column 

Number  of  Sorties  that  Land  Code  3  /Sorties  Flown  x 

100 

Monthly 

Percentage 
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732 

Fix  Rate 

A  leading  indicator  showing  how 

well  the  repair  process  is 

managed.  It  is  a  percentage  of 

aircraft  with  a  landing  status  code 

of  3  (includes  system  cap  codes  3 

and  4)  returned  to  a  flyable  status 

in  a  certain  amount  of  time  (clock 

hours).  For  example  4,8. 

See  definition  column 

Code-3  Breaks  Fixed  Within  4,  8  or  12  Hours  of 

Landing/  Total  Code-3  Breaks  x  100 

Daily 

Percentage 

744 

On-Time  Operation 

Starts 

On  time  Operation  starts 

See  definition  column 

Percent  of  Operations  started  on  time 

Daily 

Count 

766 

Utilization  Rate  (UR) 

A  measure  of  the  average  use  of  a 

system  during  a  specified  period 

of  calendar  time.  Where  usage  and 

Time  is  a  variable. 

See  definition  column 

Average  usage/time  -  flying  hrs/PAA  per  month 

(assigned) 

Monthly 

Percentage 

800 

Weapon  System 

Availability  (WSA) 

Percent  of  hours  a  reported  unit 

possessed  weapon  system  was 

capable  of  performing  any 

assigned  mission  compared  to  the 

Total  Active  Inventory  hours  for 

the  fleet.  This  calculated  by 

month  and  retained  for  trend 

analysis. 

Data  Required:  Mission  Capable 

(MC)  Hours  for  assets  with 

Possession  Purpose  Code  =  CA, 

CB,  CC,  CF,  EH,  El,  IF,  PJ,  PL, 

PR,  TF,  TJ,  ZA,  ZB  and  Total 

Active  Inventory  (TAI)  Hrs. 

Drill  down  capability  to  the 

following  levels  is  also 

required:MAJCOM  and  Base 

Sum  of  MC  Hours 

WSA-  [  -  ]X100(%) 

Sum  of  TAI  Hours 

Monthly 

Percentage 

808 

Not  Mission  Capable 

-  Supply  (NMCS) 

The  percentage  of  time  a  reported 

unit  possessed  weapon  system 

cannot  perform  any  assigned 

mission  due  to  just  supply  (lack 

of  parts).  This  calculated  by 

month  and  retained  for  trend 

analysis. 

Data  Required:  Not  Mission 

Capable  Hours  due  to  Supply 

(NMCS)  and  total  Possessed 

Hours  for  assets  with  Possession 

Purpose  Code  =  CA,  CB,  CC, 

CF,  EH,  El,  IF,  PJ,  PL,  PR,  TF, 

TJ,  ZA,  ZB. 

Sum  of  NMCS  Hours 

NMCS  =  [  - — . . ] 

X  100  (%) 

Sum  of  Possessed  Hours 

Monthly 

Percentage 

811 

Total  Not  Mission 

Capable  -  Supply 

(TNMCS) 

The  percentage  of  time  a  reported 

unit  possessed  weapon  system 

cannot  perform  any  assigned 

mission  due  to  supply  or  both 

supply  and  maintenance.  This 

calculated  by  month  and  retained 

for  trend  analysis. 

Drill  down  capability  to  the 

following  levels  is  also  required: 

MAJCOM  and  Base. 

Sum  of  (NMCS  Hours  +  NMCB 

Hours) 

TNMCS  -  [ - ]  X 

100  (%) 

Sum  of  Possessed  Hours 

Monthly 

Percentage 
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Total  Not  Mission 

Capable  - 

Maintenance 

(TNMCM) 

The  percentage  of  time  a  reported 

unit  possessed  weapon  system 

cannot  perform  any  assigned 

mission  due  to  maintenance  or 

both  supply  and  maintenance. 

This  calculated  by  month  and 

retained  for  trend  analysis. 

Drill  down  capability  to  the 

following  levels  is  also  required: 

MAJCOM  and  Base. 

Sum  of  (NMCM  Hours  +  NMCB 

Hours) 

100  (%) 

Sum  of  Possessed  Hours 

Monthly 

Percentage 

812 

Not  Mission  Capable 

-  Maintenance 

(NMCM) 

The  percentage  of  time  a  reported 

unit  possessed  weapon  system 

cannot  perform  any  assigned 

mission  due  to  only  maintenance. 

This  calculated  by  month  and 

retained  for  trend  analysis. 

Data  Required:  Not  Mission 

Capable  Hours  due  to 

Maintenance  (NMCM)  and  total 

Possessed  Hours  for  assets  with 

Possession  Purpose  Code  =  CA, 

CB,  CC,  CF,  EH,  El,  IF,  PJ,  PL, 

PR,  TF,  TJ,  ZA,  ZB. 

Sum  of  NMCM  Hours 

NMCM  -  [ - ]  X  1 00  (%) 

Sum  of  Possessed  Hours 

Percentage 

813 

Not  Mission  Capable 

-  Both  (NMCB) 

The  percentage  of  time  a  reported 

unit  possessed  weapon  system 

cannot  perform  any  assigned 

mission  due  to  supply  (lack  of 

parts)  and  maintenance.  This 

calculated  by  month  and  retained 

for  trend  analysis. 

Data  Required:  Not  Mission 

Capable  Hours  due  to  both 

Supply  (NMCS),  Not  Mission 

Capable  Hours  due  to 

Maintenance  (NMCM),  and  total 

Possessed  Hours  for  assets  with 

Possession  Purpose  Code  =  CA, 

CB,  CC,  CF,  EH,  El,  IF,  PJ,  PL, 

PR,  TF,  TJ,  ZA,  ZB. 

Sum  of  NMCS  Hours  +  NMCM  Hours 

NMCB  -  [ - ]  X 

100  (%) 

Sum  of  Possessed  Hours 

Monthly 

Percentage 

815 

Depot  Possessed 

Percent 

The  percentage  of  the  total  active 

inventory  hours  that  are  in  depot 

possessed  status.  This  calculated 

by  month  and  retained  for  trend 

analysis. 

Data  Required:  Possessed  Hours 

for  assets  with  Possession 

Purpose  Codes  =  DJ,  DK,  DL, 

DM,  DO,  DR,  and  Total  Active 

Inventory  (TAI)  Hours. 

Sum  of  Possessed 

Hours 

Depot  Possessed  %  =  [ - ]  X  100 

(%> 

Sum  of  TAI  Hours 

Monthly 

Percentage 

817 

MICAP  Hours 

MICAP  hours  are  accrued  in  a 

given  month  for  items  designated 

as  affecting  mission  capability 

that  are  on  backorder.  For  every 

day  during  the  month  the 

requisition  is  unfilled,  24  hours 

are  assigned  to  the  requisition. 

Month-to-date  (MTD)  MICAP 

hours  are  reported  by  budget 

code.  The  MICAP  information 

reported  is  a  snapshot  of  MTD 

hours  for  the  previous  month 

taken  on  the  fifth  day  of  each 

month. 

MICAP  Hours  =  [(stop  day  -  start  day  -  1)  x  24]  + 

[(24  -  start  hour)  +  stop  hour] 

Monthly 

Hours 

Total  Acitve 

Inventory 

Possessed  number  of  aircraft  by 

weapon  system 

Assets  with  Possession  Purpose 

Code  =  CA,  CB,  CC,  CF,  EH, 

El,  IF,  PJ,  PL,  PR,  TF,  TJ,  ZA, 

ZB 

Number  of  Active  weopon  system  assets 

Monthly 
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Appendix  C  Metric  Characteristic  Coding 

Process  Management  Factors 

1.  Manning  Utilization  Management 

a.  Forecasting 

b.  Dispatching 

c.  Skill  Management 

i.  Training 

ii.  Equipping 

d.  Accountability 

2.  Maintenance  Plan  Execution  Management 

a.  Response  management 

b.  Key  to  decision  making  process 

c.  MDS  maintenance  planning 

3.  Mission  Execution  Management 

a.  Mission  Forecasting  Management 

b.  Airframe  Operational  Readiness 

c.  Airframe  Asset  Scheduling 

4.  Support  Asset  management 

a.  Supply  Process  visibility 

i.  Asset  Level  Management 

ii.  Forecasting/planning 

b.  Organic  Support  monitoring 

i.  Blue  Suit 

ii.  Contractor 

c.  Contract  Execution  Oversight 

i.  Warranty  issues 

ii.  Fiscal  distributions 

5.  Mission  Design  Series  specific  applications 

a.  CAF  aircraft 

b.  MAF  aircraft 

c.  New  MDS 

d.  Old  MDS 

6.  Budget  management 

a.  Budget  validation 

b.  Budget  Execution 

Valuation  Characteristic  Factors 

7.  Data  Reporting  Value 

a.  Trends  historical  data 

b.  Failure  to  Trend  data 

c.  Specific  data  capture/application 

d. 

8.  Echelon  Value 

a.  Important  to  Senior  Level  Managers 
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b.  Key  Management  Scorecard 

c.  No  tactical  level  control/response 

9.  Clarity  Value 

a.  Clear  Definition 

b.  Similarity  to  Current  Metric 

c.  Requires  Policy  definition  to  Determine  effectiveness 

10.  Distortion  Value 

1 1 .  Redundancy  V alue 

12.  Criteria  Value 

a.  Double  Counting  of  data 

b.  Irrelevant  measurements  criteria 

13.  Specificity  Value 

14.  Process  Control  Value 

a.  Tracks  wrong  process 

b.  Fails  to  track  entire  process 

c.  Responsibility  beyond  sphere  of  control 

15.  Decision  Process  Value 

a.  Cultural  Bias 

i.  Cultural  Perception  of  Importance 

ii.  Distracter  to  other  process  measures 

16.  Understanding  Value 

a.  Unclear  language 
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Appendix  D  Commander  Authorization  Memorandum 


24  Nov  2008 


MEMORANDUM  FOR  AFIT  IRB/ASCB 
FROM:  Captain  Brian  Waller,  AFIT/ENS1 

SUBJECT :  Request  for  permission  to  interview  members  of  your  command  for  Thesis:  Integration  of  the 
Air  Force  Aircraft  Maintenance  Metrics  Program  into  ECSS. 

1 .  The  purpose  of  this  study  is  to  determine  the  optimal  translation  of  current  AF  aircraft  Maintenance 
metrics  into  the  new  ECSS  ERP  system.  The  objective  is  to  leverage  Subject  Matter  Experts  in  evaluating 
the  propose  metrics  construct  to  validate  the  translation.  The  results  of  this  study  will  result  in  a  signed  and 
publish  thesis  in  order  to  meet  the  graduation  requirements  of  AFIT. 

2.  This  request  is  to  allow  Capt  Brian  Waller  to  conduct  interviews  of  members  of  your  command.  The 
subject  of  these  interviews  will  be  to  gather  data  on  the  impressions  and  professional  evaluation  of  the 
developed  performance  measures.  All  information  collected  by  this  interview  is  for  the  sole  purpose  of  the 
above  named  thesis.  All  participation  is  voluntary  and  all  participants  will,  to  the  best  ability  of  this  study, 
be  kept  anonymous  and  all  information  will  not  be  attributable  to  any  individual. 

3.  If  you  have  any  questions  about  this  request,  please  contact  Dr.  Jeffery  Ogden,  DSN  225-3636  x 
4653/Jeffery. ogden@afit.edu,  or  Capt  Brian  Waller  (primary  investigator)  -  Phone  843-819- 
3383/brian.  waller@afti.af.mil. 


BRIAN  D.  WALLER,  Captain,  USAF 
Graduate  Student,  AFIT/ENS. 


1st  Ind,  XXXXXXXXXX 

MEMORANDUM  FOR  CAPTAIN  BRIAN  WALLER 
Permission  Granted/Declined 


“ Name  ”,  Rank,  USAF 
Commander,  XXX  XXXXXXXX 
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Appendix  E  Study  Introduction  and  Permission  Memorandum 


19  Nov  2008 


MEMORANDUM  FOR  AFRL/Wright  Site  IRB 

FROM:  Capt  Brian  Waller 
AFIT/ENS1 

SUBJECT :  Request  for  consent  to  participate  in  study  interview  for  Thesis:  Integration  of  the  Air  Force 

Aircraft  Maintenance  Metrics  Program  into  ECSS. 

1.  The  purpose  of  this  study  is  to  determine  the  optimal  translation  of  current  AF  aircraft  Maintenance 
metrics  into  the  new  ECSS  ERR  system.  The  objective  is  to  leverage  Subject  Matter  Experts  in 
evaluating  the  propose  metrics  construct  to  validate  the  translation.  The  results  of  this  study  will  result 
in  a  signed  and  publish  thesis  in  order  to  meet  the  graduation  requirements  of  AFIT. 

2.  This  request  is  for  participant  consent  to  be  interviewed.  The  purpose  of  this  interview  will  be  to 
gather  data  on  the  impressions  and  professional  evaluation  of  the  performance  measure  correlation. 

All  information  collected  by  this  interview  is  for  the  sole  purpose  of  the  above  study.  All  participation 
is  voluntary  and  no  penalties  exist  for  refusal  to  participate.  All  participants  will,  to  the  best  ability  of 
this  study,  be  kept  anonymous  and  all  information  will  not  be  attributable  to  any  individuals  and  be 
destroyed  after  a  period  of  3  years.  Participants  may  choose  at  anytime  to  decline  participation  with 
this  study. 

3.  Permission  to  conduct  this  interview  was  granted  by  “ Unit  commander” . 

4.  If  you  have  any  questions  about  this  request,  please  contact  Dr.  Jeffery  Ogden,  DSN  225-3636  x 
4653/Jeffery.ogden@afit.edu,  or  Capt  Brian  Waller  (primary  investigator)  -  Phone  843-819- 
3383/b  rian.waller@afti.af.mil. 


BRIAN  D.  WALLER,  Capt,  USAF 
Graduate  Student,  AFIT/ENS. 

I  herby  give  consent  to  be  interviewed  for  the  purpose  of  the  study  Thesis:  Integration  of  the  Air  Force 
Aircraft  Maintenance  Metrics  Program  into  ECSS. 
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Vita 


Captain  Brian  David  Waller  graduated  from  Rutherford  High  School  in  Panama  City, 
Florida.  He  was  accepted  into  Southern  Illinois  University,  Carbondale.  In  May  2001, 
he  graduated  with  a  Bachelor  of  Science  in  Architectural  Studies  and  an  Associates  of 
Applied  Arts  in  Graphic  Design  and  was  commissioned  through  AFROTC  Detachment 
205.  His  first  assignment  was  the  86th  Airlift  Wing  at  Ramstein  AB,  Germany.  He  held 
several  positions  including  Aerospace  Ground  Equipment  Flight  and  Fabrication  Flight 
Commanders,  as  well  as  76th  Aircraft  Maintenance  Unit  Officer  in  Charge.  During  this 
assignment  he  deployed  in  support  OPERATION  ENDURING  FREEDOM  as  the  C-130 
stage  maintenance  officer.  In  Sep  2004,  he  was  assigned  to  the  437th  Air  Mobility  Wing 
at  Charleston  AFB,  South  Carolina  where  he  served  as  Shark  Aircraft  Maintenance  Unit 
Officer  in  Charge,  Maintenance  Flight  Officer  in  Charge,  and  Maintenance  Operations 
Flight  Commander.  While  stationed  at  Charleston,  he  deployed  in  support  of 
OPERATION  IRAQI  FREEDOM  as  Commander,  447th  Aircraft  Maintenance  Squadron. 
In  Aug  2007,  he  entered  the  Graduate  School  of  Engineering  and  Management,  Air  Force 
Institute  of  Technology.  Upon  graduation,  he  will  be  assigned  to  the  Air  Force  Logistics 
Management  Agency  (AFLMA)  at  Maxwell  AFB-Gunter  Annex  in  Montgomery, 
Alabama. 
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Study  Participant 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  074-0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other 
aspect  of  the  collection  of  information,  including  suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information 
Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other 
provision  of  law,  no  person  shall  be  subject  to  an  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DD-MM-YYYY) 

26-03-2009 

2.  REPORT  TYPE 

Master’s  Thesis 

3.  DATES  COVERED  (From  -  To) 

Sep  2007-  Mar  2009 

4.  TITLE  AND  SUBTITLE 

Evaluation  of  Air  Force  Aircraft  Maintenance  Metrics  for 
Integration  into  the  Expeditionary  Combat  Support  System 
AFIT/GLM/ENS/09-13 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 
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Waller,  Brian  D.,  Captain,  USAF 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 
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